I ADIS International Conference on Cognition and Exploratory Learning in 
Digital Age 2012 

The IADIS International Conference on Cognition and Exploratory Learning in Digital Age was held in 
Madrid, Spain, October, 19- 21, 2012. 

The IADIS CELDA 2012 Conference intention was to address the main issues concerned with evolving 
learning processes and supporting pedagogies and applications in the digital age. There had been 
advances in both cognitive psychology and computing that have affected the educational arena. The 
convergence of these two disciplines is increasing at a fast pace and affecting academia and professional 
practice in many ways. Paradigms such as just-in-time learning, constructivism, student-centered learning 
and collaborative approaches have emerged and are being supported by technological advancements 
such as simulations, virtual reality and multi-agents systems. These developments have created both 
opportunities and areas of serious concerns. This conference aimed to cover both technological as well as 
pedagogical issues related to these developments. Main tracks have been identified. However innovative 
contributions that do not easily fit into these areas will also are considered as long as they are directly 
related to the overall theme of the conference - cognition and exploratory learning in the digital age. 

The main tracks for submissions were: Acquisition of expertise, Assessing progress of learning in complex 
domains, Assessment of exploratory learning approaches, Assessment of exploratory technologies, 
Cognition in education, Collaborative learning, Educational psychology, Exploratory technologies (such as 
simulations, VR, i-TV and so on), Just-in-time and Learning-on-Demand, Learner Communities and Peer- 
Support, Learning Communities & Web Service Technologies, Pedagogical Issues Related with Learning 
Objects, Learning Paradigms in Academia, Learning Paradigms in Corporate Sector, Life-long Learning, 
Student-Centered Learning, Technology and mental models, Technology, learning and expertise and 
Virtual University. 

The IADIS CELDA 2012 Conference received 98 submissions from more than 24 countries. Each 
submission was reviewed in a double-blind review process by at least two independent reviewers to 
ensure quality and maintain high standards. Out of the papers submitted, 29 were accepted as full papers 
for an acceptance rate of 30%; 17 were accepted as short papers and 3 were accepted as reflection 
papers. Authors of the best published papers in the CELDA 2011 proceedings will be invited to publish 
extended versions of their papers in a special issue of Computers in Human Behavior (an Elsevier journal, 
ISSN: 0747-5632), and in an edited volume to be published by Springer. 

In addition to the presentation of full papers, short papers and reflection papers, the conference also 
includes a keynote presentation from internationally distinguished researchers. We would therefore like to 
express our gratitude to Professor Norbert M. Seel, Department of Educational Science, University of 
Freiburg, Germany, as the CELDA 2012 keynote speaker. 

Keynote Lectures: 

K1 - THE THREE LEARNING SCIENCES (BIOLOGICAL, ARTIFICIAL, HUMAN) by Professor Norbert M. 
Seel, Department of Educational Science, University of Freiburg, Germany 

Abstract 

Learning is existential, and so its study must be complex and interdisciplinary. Over the past centuries, 
researchers from different fields have developed many theories to explain how humans and animals learn 
and behave, i.e., how they acquire, organize, and deploy knowledge and skills. Basically, learning is 
defined as a relatively permanent change in behavior and/or in mental associations due to specific 
experiences. Learning is a response to environmental requirements and different from biological 
maturation, which, however, is a fundamental basis for learning. Beyond psychology and biology, 
disciplines such as anthropology, sociology, and education focused on the topic of human learning in the 
course of the past centuries. However, one of the most important innovations for research on learning 
resulted from the emerging computer sciences and their focus on machine learning. Machine learning 
usuallyrefers to changes in systems that perform tasks associated with artificial intelligence (Al). Many 
techniques in machine learning are derived from the efforts of psychologists to make their theories of 
animal and humanlearning more precise through computational models. Conversely, it seems that the 
concepts and techniques being explored in the field of machine learning also illuminate certain aspects of 
the biology of learning. Accordingly, closely related to machine learning is also the study of human and 
animal learning in psychology, neuroscience, and related fields. In my presentation I will focus on the 
biological foundations of learning, mainly discussed in terms of the interplay between assimilation and 
accommodation that correspond the basic functions of schemas and mental models. Second, I will focus 
on cumulative or incremental learning as an example to demonstrate basic correspondences between 



theories of human and artificial learning. Finally, I will discuss practical implications for interdisciplinary 
research on human and artificial learning. 


The Proceedings were both published in Book (ISBN: 978-989-8533-12-8) and CD-ROM (ISBN: 978-989- 
8533-13-5) by IADIS Press. 
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REDEFINING PUBLIC POLICY EDUCATION THROUGH 
AN EXPLORATORY DIGITAL CURRICULUM: 
SINGAPORE’S STATECRAFT X 


Ming Hui Shoon and Yam San Chee 

National Institute of Education, Nanyang Technological University 
1 Nanyang Walk, Singapore 637616 


ABSTRACT 

Given the current international context of instability and uncertainty, we were driven by the desire to utilize a digital 
game to cut across the complexity of public policy, so as to educate our young with the experience and deep learning to 
be appreciative, accountable and proactive citizens of a globalized world. Elaving developed a curriculum that synthesizes 
technology, philosophy and pedagogy, we began our iterations of exploration with Singapore public schools. Through 
our project, we wanted to share our idea that the learning of public policy, in the form of Singapore’s Social Studies 
curriculum, was disadvantaged with the traditional, transmission-oriented mode of education, and had to be experienced 
and performed. Data from the impactful dialogic sessions conducted showed that students experienced realistically the 
problems faced in public policy decision making, and gained a sense of what governance, learnt performatively entailed. 


KEYWORDS 

Digital Games, Public Policy, Dialogic Learning, Performance, Citizenship, Governance 


1. INTRODUCTION 

We live today in a world beset with international complexity and danger. A panorama of the past five years 
of international affairs yields the instability of governmental systems in some regions of the world. Regions 
are affected not only by economic woes of excess, ignorance and the lack of political compromise, but also, 
the uprising of the human spirit for freedom and transparency, spreading across the vastness of continents. 
Human civilization has never been more interconnected, and has not ever had such a chance for global 
influence, if not for the speed and development of technology. This is especially so in areas like mobile 
communications and social media, avenues favored among the young, globalized citizens of a new 
prospective world order. Governments trying to harness this relentless technology in development to better 
manage their states would realize that this would not be viable without addressing the new needs of education 
for the young, in the area of public policy, through a digital game. 

With the trans-continental mobility of people, ideas and digital influences, the young, globalized citizens 
of this new world order, spurred by the ability to communicate their aspirations through social media, would 
not want to be bystanders in the shaping of their countries’ futures. Globalization itself serves as an agent for 
obligatory education reform, favoring curricula supporting global interdependence (Hershock, Mason & 
Hawkins, 2007). Hence, more so in Asia than in other regions, to facilitate their participatory rigor, 
educational development in areas like the Social Sciences might gradually shift to being a student-centered 
endeavor, aided by a means to engage in social learning. This should be the case specific at the level of 
Secondary education - the ages of 13 to 16, and involving the common subject long used to ‘teach’ concepts 
of governance. Social Studies, an amalgamation of History, Geography, Political Science, Economics and 
Sociology (Ministry of Education, 2012). 
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2. TECHNOLOGY’S ROLE IN 21 st CENTURY PUBLIC POLICY 
EDUCATION 

In a human resource dependent nation-state like Singapore, a hub connecting Western and Eastern interests, 
every learned student matters in his or her contribution towards nation building. As Singapore moves to 
become a society of further inclusiveness, the citizenry has to be educated away from the transmission-based 
mode of learning. This is strikingly so in the subject of Social Studies, taking into consideration Social 
Studies’ limitations in simply telling the students about the ‘facts’ of governance without assenting for 
performance. Allowing for performance would bridge the gap between the theories and principles of 
governance, and the actuality of enacting governance, enabling in the students a deeper plane of 
understanding and awareness. 

In this paper, we share our solution to addressing the needs of a student-centered, digital Social Studies 
curriculum so to as encourage a forward-looking vision of preparing the young of Singapore as global 
citizenry in their responses to public policy initiatives, both local and internationalized. We will discuss how 
Statecraft X, a mobile game-based learning curriculum, represents our contextualized ideals of necessity and 
philosophy of education. Following this, by examining selective instances, namely the utterances of the 
students during the dialogic sessions, we will explain how Statecraft X provided the impetus for students to 
think and enact public policy without having first learnt the ‘factual content' found traditionally in their 
Social Studies textbooks, and how this led to their incremental understanding of what governance entails. 

2.1 Literature Review 

Much of our motivation and desire to engage in the Statecraft X project was contingent with our 
understanding of a variety of literature on learning. Video games, or more appropriately in today’s mobile 
context, digital games, have immense potential as learning tools. Games are multidimensional, challenging 
and are designed to allow for learning to take place. They alter the belief that thinking is an activity that takes 
place in solitude and instead, identifies how the combination of emotion and pleasure derived from play are 
key to thinking and learning (Gee, 2007). 

In developing as well as developed societies that prize inclusiveness, approaches to learning have to 
factor in the growth of creative differences in individuals. This must be supported by the need for experience 
in environments, in the Deweyan sense of educational pragmatism, where consequence, action and thinking 
in learning come together without a mechanical, predictable end result (Elkjaer, 2009). To the extent that this 
is so, learning cannot and should not be regarded as an individual activity, and must take place in social 
contexts, taking in consideration that constructs of what we know, and how we make meanings of them are 
situated in communities (Wenger, 2009). 

Learning should not take place with an emphasis on the capacity of the students’ capabilities, but rather, 
the action being enacted, so as to ensure learning through performance, which is reflexive in nature 
(Baumann, 1989). A performance oriented model of learning is advantaged when compared to the human 
information processing model, which regards learners largely as storage machines, as performance leads to 
the creation of new meanings and understandings of existential occurrences only possible by a biological 
being (Chee, in press). That, in these perspectives explained, is why Statecraft X accentuates digital contexts 
and socialized learning with opportunities for performance. 

2.2 Game-Based Learning through Statecraft X 

Technology knows no boundaries, and in an ever-evolving educational landscape, the digital divide would 
render students who are not allowed to learn together with technological savvy in a much disadvantaged 
position. Statecraft X, being an innovative game-based learning endeavor, bridges this gap. It provides not 
only a platform, but a digital realm, a “magic circle,” well insulated from the distractions of the real world 
(Nielsen, Smith and Tosca, 2008) to allow for students to conduct personal, yet socially connected 
experiments on performing governance. The game is run on Apple iPhones as an installed application. The 
portability and island-wide availability of 3G, as well as Wi-Fi connections enhances the mobile playability 
of Statecraft X. Functionally, the game is akin to a miniaturized version of a Massive Multiplayer Online 
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Role-Playing Game or MMORPG for twenty plus players, where game play is only possible with teams of 
participants, and players navigate through town, world map and specific in-game action menus on their 
iPhone screens, like the town map shown in Figure 1. 



Figure 1. The town map where players have access to buildings and their functions. 



Figure 2. The world map where a player has entered another town and has a range of options. 

Statecraft X is set in the fantasy kingdom of Velar, which stands without a ruler after the death of a wise 
and capable king. Over the course of three weeks, students play in groups of four or five, where they 
represent factions fighting for control, influence and power over Velar, and to control the capital city 
Topezios. The four factions are Dragon, Pegasus, Phoenix and Griffin. Within each faction, each player starts 
off as a governor of a town. Statecraft is innovative in giving each player the option to vary and manipulate 
their town’s development pertaining to issues like housing, healthcare, defense and the acquisition and 
production of resources. Adding on to this flexibility is the complexity of adjustable tax rates, workers’ 
wages and even limitations on the skill level and trainability of workers and citizens who have decided to 
make a particular player’s town home, seen in Figure 3. 
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Citizens with the following Job Skills 
are allowed to enter the town: 


Farmer (Level 1 and above) 
Water Collector (Level 1 and above) 
Wood Cutter (Level 1 and above) 
Factory Worker (Level 1 and above) 
Healer (Level 1 and above) 
Infantry (Level 1 and above) 
Archer (Level 1 and above) 


« 


- - 


Figure 3. An in-game menu where players modify immigration policy. 


The social aspect of control is also not forsaken as players have a say over the holding of ideological 
rallies and beneficial multicultural events. Beyond the governorship of each individual town, players, at both 
a personal and factional level, have to make constant decisions on the organizing of diplomacy, conflict, 
influence and control over artificial intelligence controlled, and also, player controlled towns. The governors 
of the towns in Velar also meet with the random and sometimes multiple event occurrences of bandit attacks, 
epidemics and finally, towards the end of the game, an onslaught by a neighboring country, Salfreda, which 
tries to seize control of the entire kingdom. The random events that occur add to the challenge of putting in 
place strategic and decisive policies, and are reflective of the unpredictable nature of real world governance. 
At all phases of the game, students’ progress in their factions is captured daily in the form of economic and 
happiness graphs, reminding the students of the realities of having to balance societal well-being with 
economic prosperity (Chee, Gwee & Tan, 201 1). 


2.3 A Curriculum for Digital Empowerment 

Philosophically, Statecraft X’s curriculum design was influenced by the strength of educational and social 
theories. The learning of public policy and governance cannot take place through forced, textually limiting 
transmission-based materials like textbooks as the process of social science education is lost without a form 
of unhindered development, such as that put forth in Dewey’s (Dewey, 1916; 1980) exposition on naturalism. 
Essentially, public policy, in the sense of democratic education, is composed of decisional input from a 
learned mass of citizenry, and would naturally involve conflicts of inquiry. Hence, Statecraft X allows for 
students to replicate these conflicts of decisions, and the responsibility over the lives of their townspeople in 
the game, which would lead to a deeper and more inclusive learning experience (Dewey, 1938; 2008). A 
platform with performative potential is incomplete without a binding method for students to negotiate the 
meanings of governance they have just put in practice. Thus, the Statecraft X curriculum gives students the 
opportunity to engage in dialogic sessions, where meaningful, context shaped utterances in the form of inter- 
student dialogues with one another, facilitated by the teacher, embody the backward and forward creation of 
the renewed understanding of governance in practice (Bakhtin, 1981). 

Pedagogic ally, the game is driven by the Performance-Play-Dialog model of game-based learning (Chee, 
2007; 201 1), which emphasizes the feedback loop where the performance of governance is enjoyed as a form 
of play - a naturalistic development thought to originate in one’s first experience with learning, further 
developed through meaningful dialog, and repeated as beneficial practice. The students are encouraged to be 
inquisitive and experimental in their game-based performances, and to vocalize their thoughts in teacher- 
facilitated dialogic sessions which complement the digital platform. Learning is not isolated in individuals 
but takes place socially as students, in order to secure success in the governance of their towns, have to 
collaborate and cooperate to enhance the workability of their choices made. 
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2.4 Method 

Statecraft X comprises of Game-based Learning as curriculum interventions in Social Studies classes with 
fifteen year old students. The objective of enacting the interventions was, as a research project, to enhance 
classroom teaching and learning in the upper Secondary subject of Social Studies, and investigate its 
applicability in Singapore’s context of education. This paper focuses on the curriculum interventions, and 
dialogic sessions that took place in two public schools at the Secondary level, involving six classes and four 
teachers, in the first half of 2012. The two schools volunteered to be involved in the research project as they 
were keen to have their teachers acquire new methods of facilitation, while exposing their students to digital 
learning. 

The first school had the flexibility to experience two cycles of intervention, and the second school only 
had the time in their school curriculum to fit in one cycle of intervention. The six classes comprised of four 
classes from one school and two classes from the other, making up approximately one hundred and twenty 
students. These students were from the Express stream, which forms the broad mass of students in Singapore 
Secondary schools. The teachers involved are named teachers A, B, C and D in respect of their anonymity in 
our study. Each Statecraft X class had approximately twenty students, and each curriculum intervention took 
place over three weeks, where students loaned iPhones from the research team and played the game. Within 
the intervention period of three weeks, each class underwent four dialogic sessions lasting sixty minutes 
each, which is the largest block of time allocated for Social Studies in these two schools. Audio-visual 
equipment was set up to record the interactions between the students and the teacher’s facilitation. The 
interpretations in this paper are based on the qualitative data of students’ utterances captured from the 
dialogic sessions, which were then transcribed and coded categorically based on grounded theory (Corbin & 
Strauss, 2008). 

2.5 Results and Discussion 

The richness of utterances captured in meaningful dialogue is often underestimated in its qualitative value. In 
fact, students' development in their understanding of governance became apparent when we investigated the 
utterances captured through video recordings during each of the four dialogic sessions carried out in the 
schools that collaborated with us. Taking a grounded stance, we discovered that these fifteen year old 
students, through their playing of Statecraft X, were able to reflect and express through their dialogue that 
they have understood, with significance to their level of appreciation, accountability and pro-activeness in 
four areas of public policy, without any exposure to conventional Social Studies materials. These four areas 
were: the realities of governance, social policy, economic policy and foreign policy. Examples were 
numerous and for the purpose of this paper, we will be highlighting selective instances of developmental 
realization among these students. 

2.5.1 Appreciating the Realities of Governance 

Generally, dialogue from the dialogic sessions was evident in showing a raised awareness in students’ 
appreciation of how it was a difficult task to govern. When teacher A challenged her students to make a 
choice between putting a greater emphasis on the economic development or a country’s defense needs, she 
was met with a diversity of responses, such as “It is important to be prepared in both defense and the 
economy,” and "If you have the money you can make people happy and you also have stability.” Some 
students realized that it was more important to have a balanced development, while others took an economy- 
first perspective. Fundamentally, it became clear to students that they had to be accountable for their choices 
as governors. In another session, one of teacher A’s students, having explored the various towns in Velar, 
remarked that: “When you have a big country and you have an order, it will take some time for the order to 
be passed down.” Without having been told about the difficulty of effective communication in big countries, 
this student comprehended the challenge on his own. Teacher C asked students if leaders could neglect their 
people and there was unanimity in their response that this was a possibility. Students, through this 
recognition, were aware that government leadership can and will go wrong from time to time. 

Students uncovered the paradox of governance when teacher C inquired about the students’ understanding 
of the importance of citizenry, and one of her students remarked that citizens were not only needed to provide 
for defense, but they could also extractively give a governor and his or her town money. Governance, to this 


7 



ISBN: 978-989-8533-12-8 © 2012 IADIS 


student here was not an altruistic activity but one of situational advantages. This student realized that 
governments, in ensuring the survivability of their nation, as well as their own positions as leaders, could not 
always take a selfless stand, and when the opportunity arose, had to balance themselves with some form of 
gain, possibly like the monetary gain that governors in Statecraft X experience. Whenever governors in 
Statecraft X take actions using their Action Points (AP), they would have to wait an hour for their APs to be 
replenished. Without making this overtly known to the students so as to enhance the self-exploratory element 
of the game, students recognized for themselves this limitation and in Teacher D's class, admitted that “the 
one hour gap between different plays show that we may not be able to give people what they want as fast as 
they expect it.” Statecraft X, to this student gives the “opportunity to think hard about the actions you take, 
you can strategize” and “gives us the timing to plan, real life also cannot plan like that” which is indicative 
again of the realities of governance. 

2.5.2 Understanding Social Policy with Accountability 

Interestingly, responses to social policy made up the bulk of students’ meaningful dialogue, which shows that 
despite their age, they are accountable to their social surroundings, both within the game and beyond. 
Perhaps, due to their close proximity to society itself as middle class students from public schools, this was 
their primary area of focus. Teacher A referred to an in-game event where an epidemic hit the towns of Velar 
and many students found their citizens dying from this epidemic. Students responded that some of their 
factions were well prepared for this event, as some of them had prioritized the building of a healing center, 
while others were not prepared as they were too preoccupied in the building of their barracks when the 
epidemic hit. With further dialogue, students were appreciative of the effects that the ignorance of 
governmental action could bring to society. In the game, governors can travel to other towns where they can 
hold rallies or even attempt to take over the town by force. When a student from teacher B’s class attempted 
to conquer another student’s town forcefully, he was met with failure. This student had made his attempt with 
the assumption that the use of force was the only known way to conquer another town, and had his 
assumption proven inaccurate, which became a learning point when he tried to do so. Responding to this 
episode, a student remarked to the victim that the aggressor “tried to take over your town but people drove 
him out.” Reflective of reality, the responding student realized that the strength of citizens’ belief in their 
governors and the hope that it brought can overcome invasive political maneuvers. 

Of course, the reverse could happen and it certainly did. In one of teacher C’s class, a student was 
preoccupied with conducting rallies purely to “increase people’s trust so that the person can take over others’ 
town” and elaborating on that “increase people’s trust in the way of governance leading to people trusting 
you more and the people want you to govern instead of using force and violence.” This student saw social 
policy in connection with diplomacy in the expansion of governmental influence, but at the same time 
recognized the potential for soft power application to conquer without violence. Placing a strong emphasis on 
the citizenry's belief, a student from teacher C’s class exclaimed that “if there is no belief in the governor the 
town cannot function.” Students from both classes therefore understood that for societies to function 
harmoniously, and for the citizenry to provide the best of their talents and capabilities for continued growth 
and prosperity, there had to be belief in governance, a reality of many political contexts. Teacher A quizzed 
her students on their thoughts and feelings about the immigration of talents into a country, drawing on the 
Statecraft X in-game function where governors could restrict the type of immigrants as well as the skill level 
of the immigrants. This led to a peculiar, alternative pro-immigration response when a student chose not to 
echo the other responses of anger but rather that foreign talents would allow for “the need to continue to 
improve yourself.” Armed with dialogue from both sides of the immigration debate, the students emerged 
from the dialogic session more appreciative of foreigners in their country and accountable for their criticisms. 

2.5.3 Experience and Thinking about Economic Policy 

The dialogic sessions revealed that despite the lack of ‘knowledge’ in economics, students could reflectively 
suggest ideas relating to tax and labor policies. This would not be possible without invoking their higher- 
order thinking abilities, and was due to them experiencing economy-related policies at first hand. When it 
came to the management of workers, students from teacher A’s class were in favor of giving opportunities for 
gradual salary increments and in their deployment, they stressed the need for not having excessive people 
employed as to minimize the loss of revenue and profit. 
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In another session, students from teacher A’s class also observed, through their manipulation of the tax 
rates in the game and the subsequent responses of the citizens' happiness levels, that a reduction of taxes and 
increases in the citizenry’s happiness as not something that occurred across the board. With concern to 
managing a country's reserves and budgeting, one of teacher C’s students explained: “save money in case the 
food runs out, there are illnesses or war, where food prices will go up,” and “in times of war increase taxes so 
that there is more money to train soldiers and high tech weapons.” This student was lucid about having strong 
reserves to cater for times of crisis and the necessity of increasing taxes to suit a more immediate concern 
threatening a country. The productivity of labor is another concept explored through students’ responses. 
Teacher B questioned the students about how they trained workers specific to the water tower and was given 
the reply that the workers would be “more efficient, have a higher productivity and more output of water.” 
Students were proactively concerned about the running of their towns because they could, through play, 
intervene as the government and shape their own polity. 

2.5.4 Engaging in Foreign Policy with Pro-activeness 

Statecraft X allowed for students to experience what it was like enacting and engaging in policy without 
learning about the concepts first. This led to them showcasing their prowess in dialoguing about military 
strategy, the act of diplomacy and also the interdependence of nations, in many instances with pro-active 
engagements. In the aftermath of a series of Salfredan attacks, a student in teacher B’s class found that he had 
lost five out of his seven towns to forces from Salfreda, and thinking about this situation led the student to 
conclude that he was managing too many towns. This is in actuality, a common problem of big countries, 
which have centralized governments and defending these countries' borders is no easy feat. 

Dialoging about military expansion and action resulted in students unveiling the various reasons for war, 
and not to view war merely as an act of aggression. Students in teacher B's class found that governors 
declared war with one other to “get more resources, to get more land for the citizens,” in the words of one 
student. Students also realized the importance a strong defense plays in deterring invasions and keeping the 
citizenry happy. One of teacher B's students found that her town's wall defense was zero, and deduced that 
as a result of this, there was “no stability, no safety and no sense of security” which led to the town's people 
being unhappy despite racial harmony functions being organized for their benefit and to keep them happy. In 
teacher D's class, the perils of a small, successful nation was brought out when a student raised the point that 
as a town, representative of small countries in reality, “when you’re outstanding you'll be the center of 
attraction and there will be those who wish to harm you and those who wish to look out for you,” which 
highlights the importance of necessary alliances as well as caution in diplomatic relations, and the dangers of 
invasion. 


3. CONCLUSION 

In this paper, we argued that there is no better time than the present for education to emphasize the learning 
of public policy, given the turbulence in international affairs and the growing influence which globalized 
digital citizens have over matters of resolution. Being inspired for a noteworthy solution, we began the use of 
exploratory technology, manifested in a digital game. Statecraft X, to allow students to experience, through 
the PPD model of game-based learning, the complexity of public policy as a governor of a town in a fantasy 
realm. Through the examination of our research data, notably the utterances from the dialogic sessions, we 
found developmental evidence of students coming to terms with concepts of governance without having had 
any formalized ‘teaching' sessions during their Social Studies classes. 

The curriculum interventions were successful, as identified in the categorization of four areas of public 
policy concerns: the realities of governance, social policy, economic policy and foreign policy. Students were 
appreciative of the difficulties of governance, could conduct social policy with accountability, run economic 
policy with higher-order thinking and engage in foreign policy pro-actively by the end of each of the three 
week cycles. Certainly, the implementation of Statecraft X in schools faced a number of challenges. As 
Singapore schools ran on closely planned schedules, we were unable to run the curriculum together among 
the schools at the same time, and among the students involved in the curriculum intervention, there were a 
number who tended not to contribute sufficiently to the dialogic data in the sessions they had. However, these 
concerns can be addressed as the teachers we worked with for this series of iterations are now experienced in 
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both the game -based and dialogic pedagogy, and exhibit the increased ability to facilitate towards the 
completeness of our intended scenarios. We are therefore confident, that further iterations of the Statecraft X 
curriculum and its refinements from this series of iterations would allow for the curriculum to encompass a 
wider range of concepts related not just to governance but that of a broader conceptual framework which 
would allow students to emerge as learners ahead of the demands of the 21 st century. 


REFERENCES 


Book 

Bakhtin, M. M., 1981. The dialogic imagination: Four essays. University of Texas Press, Austin, Texas. 

Baumann, R., 1989. Performance. In E. Barnouw, ed. 1989. International encyclopedia of communication. Oxford 
University Press, New York. 

Chee, Y. S., in press. Video Games for "Deep Learning”: Game-Based Learning as Performance in the Statecraft X 
Curriculum. To appear in Lee, C. B. and Jonassen, D. H., eds. Fostering conceptual change with technology: Asian 
perspectives. Cengage Learning, Singapore. 

Corbin, J. and Strauss, A., 2008. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded 
Theory (3 rd Edition). Sage, London. 

Dewey, J., 1916; 1980. Democracy and education (Vol. 9, John Dewey: The Middle Works, 1899-1924). Southern Illinois 
University Press, Carbondale, Illinois. 

Dewey, J., 1938; 2008. Experience and education, Freedom and culture, Theory of valuation, and Essays (vol. 13, The 
Later Works of John Dewey, 1925-1953). Southern Illinois University Press, Carbondale, Illinois. 

Elkjaer, B., 2009. Pragmatism: A learning theory for the future. In Illeris, K, ed. 2009. Contemporaiy theories of 
learning. Routledge, London, United Kingdom. 

Gee, J. P., 2007. Good Video Games + Good Learning: Collected Essays on Video Games, Learning and Literacy. Peter 
Lang, New York. 

Hershock, P. D., Mason, M. and Hawkins, J. N., 2007. Changing Education: Leadership, Innovation and Development in 
a Globalizing Asia Pacific. Comparative Education Research Centre. Hong Kong. 

Ministry of Education, 2012. Combined Humanities GCE Ordinary Level (Syllabus 2192). Author, Singapore. 

Egenfeldt-Nielsen et al., 2008. What is a game? In Nielsen, S. E., Smith, J. H. and Tosca, S. P., 2008. Understanding 
video games: The essential introduction. Routledge, New York. 

Wenger, E.. 2009. A social theory of learning. In Illeris, K, ed. 2009. Contemporaiy theories of learning. Routledge, 
London, United Kingdom. 

Journal 

Chee, Y. S., 2007. Embodiment, embeddedness and experience: Game-based learning and the construction of identity. 
Research and Practice in Technology Enhanced Learning, 2(1), 3-10. Doi: 10. 1142S/S 1793206807000282 

Chee Y. S., Gwee, S. and Tan, E. M., 2011. Learning as become citizens by enacting governorship in the Statecraft 
curriculum: An evaluation of learning outcomes. International Journal of Gaming and Computer Mediated 
Simulations, 3(2), 1-27. 

Chee Y. S., 2011. Learning as becoming through performance, play and dialog: A model of game-based learning with the 
game Legends of Alkhimia. Digital Culture and Education, 3:2, 98-122. 


10 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


E-LEARNING 3.0 = E-LEARNING 2.0 + WEB 3.0? 


Fehmida Hussain 

Middlesex University, Dubai 


ABSTRACT 

Web 3.0, termed as the semantic web or the web of data is the transformed version of Web 2.0 with technologies and 
functionalities such as intelligent collaborative filtering, cloud computing, big data, linked data, openness, interoperability 
and smart mobility. If Web 2.0 is about social networking and mass collaboration between the creator and user, then the 
Web 3.0 is referring to intelligent applications using natural language processing, machine-based learning and reasoning. 
From the perspective of advancements in e-Learning, the Web 2.0 technologies have transformed the classroom and 
converted a passive learner into an active participant in the learning process. This paper posits that the way both previous 
generations of e-Learning (1.0 and 2.0) have emerged with the prevalent technologies in their kin Web versions (1.0 and 
2.0, respectively), it can be argued that e-Learning 3.0 will provide all earlier generations’ capabilities enhanced with the 
Web 3.0 technologies. Furthermore, in this paper, reviewing all the theories of learning and examining closely the theory 
of connectivism (considered to be the theory of learning for the digital age), it is argued that since most of the 
technologies that are to be a part of e-Leaming 3.0 are addressed by these principles, a call for a new learning theory for 
e-Learning 3.0 is not justified. Finally, a review of the secondary literature shows that there will be various challenges 
and issues related to prevalence and adoption of e-Learning 3.0 technologies, for example increased privacy and security 
risks, web accessibility, readiness of the users, requirement for further standardization of e-Learning technologies and 
social issues in term of increase of the digital divide. 


KEYWORDS 

e-Learning 2.0, e-Learning 3.0, Web 2.0, Web 3.0, Learning theories, educational technology. 


1. INTRODUCTION 

E-Learning 3.0, is it the way ahead in educational technology or just another buzz word in the digital-spaces 
and education? At a prestigious IEEE conference, an interesting paper was published titled “ E-Learning 2.0 
= e-Learning 1.0 + Web 2.0” (Ebner, 2007, p. 1235)? After five years of incredible technological 
advancements, a reappraisal of the digital spaces is due; it is only fair to ask the question: Is E-Learning 3.0 
= E-Learning 2.0 + Web 3.0? Exploring what entails e-Learning 3.0, there are three parts of this paper: (1) 
How Web 3.0 technologies will be incorporated in the existing e-Leaming 2.0? (2) Is there a need for a new 
learning theory for e-Learning 3.0? (3) What are the challenges and issues related to adoption of e-Learning 
3.0? 

This paper first describes the three generations of Web, usually referred to as, the Web 1.0, Web 2.0 and 
the recently used neologism, Web 3.0. After explaining how the educational technologies and the information 
systems used in each are different, these are related to the concept of e-learning. E-Learning literature also 
broadly distinguishes between e-Learning 1.0, e-Leaming 2.0 and e-Learning 3.0; hence the paper draws a 
parallel between the generations of the Web and e-Leaming. 

Next, in relation to e-Learning 3.0, as education researchers are questioning (Wheeler 2009b; Meichel, 
2009) whether a new learning theory is required for e-Learning 3.0; so in the light of the existing learning 
theories, a preliminary attempt is made to address this. This paper does not indulge in the debate of whether 
connectivism is a theory or a phenomenon, nor does it go into responding to the critique that has been done 
about it. The simple approach taken in this paper is to make the basis of the argument all the important 
principles of the connectivist theory of learning and then examine how the technological shift maybe 
supported by these principles. 
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Nevertheless, with any advancement and more so, the technological advancement, there are associated 
challenges in terms of technological, ethical and social issues which are briefly discussed and highlighted in 
the end of the paper. 

2. HOW WEB 3.0 TECHNOLOGIES CAN BE INCORPORATED IN THE 
EXISTING E-LEARNING 2.0? 

2.1 Evolution of the Web 

This section briefly covers the three generations of the Web in terms of the capabilities, technologies and its 
usage. The Web has evolved from the early days of the ENQUIRE project to the transformation of Web 3.0 
(Berners-Lee et al., 2001; Berners-Lee 1995) Broadly speaking, where the Web 1.0 connects real people to 
the world wide web (www), the Web 2.0 connects real people who use the www, the Web 3.0 will connect 
the virtual representatives of the real people who use the www. So, it is believed that Web 1.0 is about 
providing information, Web 2.0 is about overload of information and the Web 3.0 is about control of 
information (Rego, 2011). 

Web 1.0 is generally referred to as the “read-only Web” making content available online for viewing. 
Authors of the web generally write what they want others to view and then publish it online. The reader can 
visit these web sites and can contact the writer or publisher if contact information is available. There is no 
direct link or communication between the two. Examples of these are static web sites and web pages created 
using HTML. (Rubens et al., 2011) 

The term Web 2.0 is usually associated with the O'Reilly Media 2.0 conference (O'Reilly, 2004), but was 
actually used for the first time in early 1999. (DiNucci, 1999) As opposed to the Web 1.0 which is referred to 
as the static web, Web 2.0 is considered as the dynamic web. The users can read, write and collaborate to a 
certain extent. The latest technologies used on client side or server side in Web 2.0 are Ajax (Asynchronous 
Javascript), XML (Extensible markup language), Adobe Flash, PHP, Per, Python, Flash and so on. 

The technologies and concepts related to the Web 3.0, though still in the infancy stage, are advancing 
quite rapidly. The Web 2.0 has given rise to silos of data being generated by social networking and there will 
be a need to enable the utilization of this data. An astounding statistics by the Forrester Research (2006) 
shows that 97% of the users never look beyond the top three search results when they are searching on the 
internet. The main features of the Web 3.0 technologies which differentiate it from its earlier generation, 
Web 2.0 are given as follows (Cho, 2008; Wheeler 2009a; Berners-Lee, 2001; Morville, 2005; Semweb, 
2011): 

• Intelligent/semantic Web: The term semantic web refers to the W3C’s vision on the Web of linked data 
enabling people to create data and build vocabularies. .Simply put, semantic web is all about describing 
things in a form that is understood by computers 

• Openness and interoperability: This refers to openness in terms of application programming interfaces, 
data formats, protocol and interoperability between devices and platforms. 

• Global repository of data: This is the ability of information to be accessed across programs and across 
the web. 

• 3D Virtualization: Extensive use of 3D modeling and 3D spaces using services like second life and 
personalized avatars connected to your devices. 

• Distributed and Cloud Computing: The delivery of computing as a service rather a product. 

Al and machine learning are the main driving force behind the Web 3.0. For example, in Web 2.0, 
searching the word e-Learning on Google will give a plethora of unrelated hits, but the Web 3.0 will solve 
this problem by providing context to searching for online information. As the Web 3.0 is also referred to as 
the Semantic Web of Data (Berners-Lee Video), there will be huge datasets created, so the need of the time is 
management of ‘Big Data’ and ‘Linked data’ (Fischetti, 2010). The Web 3.0 will make use of technologies 
such as RDF (Resource Description Framework, SPARQL (Query Language for RDF), OWL (Ontology 
Web Language and SKOS (Simple Knowledge Organization System) (W3CSW, 2009); these will help 
structure information such that programs like web spiders and web-crawlers can search, discover, collect and 
analyze information from the web (RDF, 2004). “If HTML and the Web made all the online documents look 


12 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


like one huge book, RDF, schema and interface languages will make all the data in the world look like one 
huge database”, (Berners-Lee, 1999). 

2.2 Evolution of e-Learning 

E-learning is usually understood as instruction delivered via a computer in teaching and learning. A number 
of other terms are synonymously used with e-Learning for example, computer based training, online learning, 
virtual learning, web-based learning and so on. The central idea is that all these refer to use of information 
and communication technology that pertain to all educational activities either performed individually or in 
groups, working online or offline, synchronously or asynchronously, via networked or standalone computers 
or other electronic devices. (Romiszowski, 2004; Garrison and Anderson, 2003) The learner of the future is 
totally digitalized. The often so-called digital natives, digital immigrants, net-generation. Generation @ are 
some of the names given by researchers to the students of today. (Prensky, 2001; Tapscott, 1997; Pelevin & 
Bromfield, 2002) 

2.2.1 E-Learning 1.0 

With the advent of the Web, the major change was to have content available online. The concept of “learning 
objects” came into being which were used to create what is known as the learning (content) management 
systems (LMS or LCMS) developed to support study management and course organization for learners. This 
is considered more of a traditional, rather than a hierarchical way of learning where communication is mono- 
directional. In this direct-transfer model, the instructor is the distributor of learning material in a media-rich 
way and addresses learners through various communication channels. This era is usually referred to as e- 
Learning 1.0. 

2.2.2 E-Learning 2.0 

Stephen Downes described the use of Web 2.0 technologies for teaching and learning as ‘e-Learning 2.0’ 
(2005). Web 2.0 for example has transformed the class room in terms of how it is not only socially but 
collaboratively constructed by using wikis, blogs, podcasts, and other social web tools. Such tools demand 
dynamic content generation which may comprise of reflections and conversation, hence requiring a 
collaboration and interaction. (Richardson, 2009) This is a collaborative way of learning where 
communication is multi -directional where knowledge may be socially constructed. 

2.2.3 E-Learning 3.0 

Education researchers are now quite freely using the term e-Learning 3.0 in various blogs and discussion 
forums. (Walters, 2010; Moore, 2010, Wheeler, 2009a) Emergence of cloud computing and availability of 
new technologies such as collaborative intelligent filtering, increased and reliable data storage capacity, 
higher screen resolutions, multi gesture devices and 3D touch user interface is leading us into the next 
generation of e-Learning. One of the big things of third generation of e-Learning will be the ubiquitous 
access to learning resources with the use of mobile devices to virtually access anything, anytime and 
anywhere. (Baird 2007; Wheeler 2009a) Technology people also suggest the use of AI and data mining for 
building the e-Learning 3.0 systems which have the capability to sift and sort big data, in turn provide to the 
learner a deeper and better understanding of the learning process itself. (Rubens et al., 2011) Further, 
education researchers believe that the underlying concept of ‘anytime, anywhere and anybody’ will be 
supported by ‘anyhow’ which will be provided by virtual 3D worlds such as use of Second Life and personal 
avatars. (Baird 2007; Rego 2010) With well-established Web 2.0 technologies and moving ahead to Web 3.0, 
research communities are talking about the personal learning environments (PLEs also referred to as mash- 
ups). “Personalization is seen as the key approach to handle the plethora of information in today’s 
knowledge-based society.” (Ebner et al., 2011, p. 22) 

Though e-Learning 3.0 systems are not prevalent commercially, however researchers are proposing such 
solutions as proof-of-concepts or working prototypes. One of the first online services to use semantic web 
automatically and intelligently organizing information about users’ specific interests is Twine. (Spivack's, 
2010) Another example of a true e-leaming 3.0 system is AHKME (Adaptive Hypermedia Knowledge 
Management E-Learning Platform) an e-Learning Information System having learning requirements 
compliant with a Web 3.0 philosophy (Rego, 201 1). 
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2.3 Relationship between Web Evolution and E-Learning Evolution 

Relating the generations of the Web with generations of e-Learning, if Web 1.0 is the read only web and Web 
2.0 is the read/write web, then Web 3.0 is the read/write/collaborate web. E-Learning 1.0 is about providing 
to the learner information access, whereas e-Learning 2.0 in addition to all e-Learning 1.0 capabilities 
provides authoring and interacting capabilities to the learner. Furthermore, e-Leaming 3.0, enabled and 
enriched with Web 3.0 technologies will promote intelligently collaborative, rich 3D virtual learning 
environments which will bring learners together for anytime, anywhere, anyhow learning experience, 
utilizing the semantic capabilities to parse the global databases of knowledge. Hence it can be deduced that 
e-Learning 3.0 will provide all earlier generations capabilities enhanced with the Web 3.0 technologies. The 
relationship between the generations of the Web and e-Learning are summarized in Table 1. 

Ebner (2007) while attempting to answer whether e-Learning 2.0 is just a sum of e-Leaming 1.0 and Web 
2.0 technologies posited that the human factor is also important, that is the readiness and acceptability of 
users towards a new technology. Therefore, Ebner (ibid) revised his equation to be e-Learning 2.0 = f (e- 
Learning 1.0, Web 2.0, human factor). In the context of this paper, when accepting Web 3.0 technologies for 
e-Learning 3.0, the human factor will also play some role but possibly not a significant one because the 
transition for the aforementioned ‘digital natives’ will be much quicker as compared to the earlier 
generations, the ‘digital immigrants’. 


Table 1. Relationship between generations of Web and e-Learning (adapted from Rego, 2011). 



Web 

E-Learning 

Version 

Concept 

Technologies 

Concept 

Technologies 

1 

Read-only or write 
only, security, web of 
documents 

HTML, HTTP, URL 

Content management, 
Unidirectional activities 

CBT, LMS, eBooks 
VLEs, 

2 

Re ad/ write 
Social web 

Dynamic web 

technologies. ASP, 

AJAX, podcasts, SNS 

Blended learning, 

content authoring. 
Bidirectional activities. 
Multimedia content 

LCMS, social networks, 
video conf ,VLEs, 
Mashups 

3 

Re ad/write/reque st/col 
laborate big data, 
linked data 

RDF, XML, OWL, 3D, 
second life 

Learner-centered, 
U-learning, knowledge 
representation 

PLEs, Social semantic 
web, second life, 

personal avatars 


3. DO WE NEED A LEARNING THEORY FOR THE WEB 3.0? 

Information on the fly and on the go has become a norm for the digital natives. (Oblinger & Oblinger, 2005; 
Oblinger, 2005 ) One relevant question to ask here is, does all the advancement in technology signal a change 
in the learning behavior, or do we need a new learning theory for the digital natives? Recently, education 
researchers are also discussing whether connectionism is adequate as a theory in terms of its coverage of the 
need for the digital age; Connectionism is so Web 2.0 (Miechel, 2009; Wheeler, 2009b). George Siemens’ 
Connectivist approach to learning in a digital age might actually be superseded by our need to re- 
conceptualize the whole idea of what learning will mean - especially when we are immersed in a world of 
ambient mobile pervasive communication where intelligent agents and filtering tools do our bidding for us. 
Based on review of the literature of learning theories, this paper makes a preliminary attempt to answer this 
question. 

Learning theory refers to a framework that helps us think about how and why change (in learning) occurs 
(Smith, 1999). A review of the literature shows that there are different orientations and approaches to 
explaining how this process of learning takes place, for example, behaviorist, cognitivist, humanistic, 
social/situational and the connectivist approaches to learning. Broadly speaking, in the education literature, 
there is reference to four theories of learning namely Behaviorism, Cognitivism, Constructivism and 
Connectivism. In Behaviorism knowledge is perceived as facts that can be transferred from teacher to student 
(can be related to e-Learning 1.0) Cognitivism opens up the black box of the mind, considering the learner as 
an information processor whereas Constructivism suggests that learners create knowledge as they try to make 
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meaning of their experiences. Connectivism, considered to be the learning theory of the digital age, according 
to Siemens (2004) is, “a successor to behaviorism, cognitivism, and constructivism.” These theories of 
learning are briefly described in Table 2, in terms of the view of the learning process, locus of learning and 
purpose of education; Table 2 is adapted from Ashworth et al., (2004) adding a column to relate with the 
connectivist approach (Siemens, 2004). 


Table 2. Summary of learning theories (adapted from Ashworth et al., 2004; Seimens, 2004). 


Theories of Learning 

Aspect 

Behaviorist 

Cognitivist 

Constructivist 

Connectivist 

Learning 

theorists 

Skinner, Pavlov, 

Bruner, Kohler, 
Piaget 

Bandura, Vgotsky 

Siemens, Downes 

View of the 

learning 

process 

Change in behavior 

Internal mental 
processes 

Construction of 
meaning from 
experience 

Connecting 
specialized 
information sets 

Locus of 
learning 

Stimuli in external 
environment 

Internal cognitive 
structuring 

Internal 
construction of 
reality by 
individual 

Draw information 
outside of our 
primary knowledge 

Purpose of 
education 

Produce behavioral 
change in desired 
direction 

Develop capacity 
and skills to leant 
better 

Construct 

knowledge 

Ability to 
synthesize and 
recognize 
connections 


Connectivism applies ideas from biological models of the brain to neural networks in machine learning; 
stating its basic principles as follows (Siemens, 2004): 

■ Learning and knowledge rests in diversity of opinions. 

■ Learning is a process of connecting specialized nodes or information sources. 

■ Learning may reside in non-human appliances. 

■ Capacity to know more is more critical than what is currently known 

■ Nurturing and maintaining connections is needed to facilitate continual learning. 

■ Ability to see connections between fields, ideas, and concepts is a core skill. 

■ Currency (accurate, up-to-date knowledge) is the intent of all connectivist learning activities. 

This paper does not indulge in the debate of whether connectionism is a theory or a phenomenon or for 
that matter does not attempt to counter some of the criticisms about it. The simple approach taken in this 
paper is to examine the important principles of the connectivist theory of learning as stated by Siemens 
(2004) and then look at the new technologies which will be introduced as a result of the advancements in the 
web technologies, thus compare and relate which technological shift may be supported by the principles of 
the connectionist learning theory; this is demonstrated in Table 3. Based on the relationship illustrated in 
Table 3, the paper posits that the cutting edge technologies to be a part of e-Learning 3.0 are adequately 
supported by the principles of the learning theory of connectivism. Hence a call for a new learning theory for 
e-Learning 3.0 is probably not justified. 
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Table 3. Web 3.0 technologies supported by basic principles of connectivism. 


Web 3.0 technologies used in e-Learning 3.0 

Basic principles of connectivism 

Social semantic networks, openness and 
interoperability 

Learning and knowledge rests in diversity of 
opinions. 

Big data or global data repository, linked data, cloud 
computing, extended smart mobile technology 

Learning is a process of connecting specialized 
nodes or information sources. Currency of 
knowledge is important. 

Machine learning, artificial intelligence, personal 
avatars, 3D visualization and interaction 

Learning may reside in non-human appliances. 

Semantic web, control of information 

Capacity to know more is more critical than what is 
currently known 

Semantic web, collaborative intelligent filtering 

Ability to see connections between fields, ideas, and 
concepts is a core skill. 

Semantic web, collaborative intelligent filtering 

Nurturing and maintaining connections is needed to 
facilitate continual learning 


4. CHALLENGES AND ISSUES RELATED TO E-LEARNING 3.0 

A review of the secondary literature shows that various challenges and issues may arise due to prevalence 
and adoption of e-Leaming 3.0 technologies. In the MashUps of the interconnected world in the hyperspace, 
there will be an increased risk of privacy of data due to difference in privacy laws from country to country. 
Also, added security risks may be faced due to lack of server-side checks and excessive privileges. (Weippl, 
and Ebner, 2008) Researchers are concerned that the ethical dilemmas will also be enhanced as due to the 
nature of the semantic web, vastness, vagueness, uncertainty and inconsistency may add to privacy and loss 
of control. (Alves, et al., 2011; Alkhateeb et al., 2010) Similar to other web-based applications, there is going 
to be more of a challenge to provide accessibility with web content to people with special needs. There have 
been some initiatives from the World Wide Web consortium on this (WAI, 2009) but much more work will 
be required in this direction. There are researchers who are apprehensive about the use of such advanced 
technologies. “There are teachers who are still struggling with Web 1.0 let aside the students. So the 
questions or apprehensions on the other side are: are our students truly ready to be autonomous learners? Are 
our teachers willing to give into technology? (Wheeler, 2009a)” As suggested earlier in section 2.3, the 
human factor is definitely a variable in the equation; the impact should reduce as the users become more and 
more comfortable with technology. Lack of standards is another concern. This is in relation to exchange of 
data and content between systems. Standard that do exist will have to be enhanced for the future generation 
of the web for example, sharable course object reference model (SCORM), IEEE learning technology 
standards committee (LTSC), Instructional Management Systems project (IMS) and so on. Educational 
technology standardization movement has become an important force, for example IMS Global Learning 
Consortium (IMS, 2011), IEEE (IEEELOM, 202), Dublin Core (DC, 2010) which are basically working 
towards standardization of educational technologies for e-Learning applications such as metadata, digital 
repositories, and many more. 


5. CONCLUSION 

Keeping pace with the unprecedented innovations in technology, the education domain has also evolved. Use 
of technology in education has had a significant impact on teaching and learning. E-Learning systems have 
also evolved with the evolution of the Web and the new technologies will continuously enable the application 
of learning and teaching theories into e-Learning practice. This paper explores how Web 3.0 technologies 
will entail e-Leaming 3.0; whether the need for a new learning theory is called for with the use of the Web 
3.0 in educational technologies and what are the potential challenges and issues related to the advent of 
e-Learning 3.0. The evolution of e-Learning (e-Learning 1.0, e-Learning 2.0 and e-Learning 3.0) is related to 
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the three generations of the Web (Web 1.0, Web 2.0 and Web 3.0). Web 2.0 and the associated technologies 
are well established and accepted by the user and the prevalence of these in e-Learning is common. This 
paper posits that, just like its predecessor, Web 3.0 technologies, once stable and well developed will further 
transform the e-Learning discipline. However, it does not seem that there is a need to call for a new learning 
theory as the theory of connectivism should be adequate. However, with the advent of any technology and its 
adoption, use of Web 3.0 and e-Learning will come with a plethora of technological, social, legal and ethical 
challenges. So the equation in the title can be modified to E-learning 3.0 = / (E-learning 2.0, Web 2.0, other 
factors and challenges), (here f means a function of) . Advanced technologies will continue to play a central 
role in the development and evolution of e-Learning; however it will do so more in the background providing 
connections between knowledge; so technology is not merely an enabling tool in education, rather a driver of 
change. 
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ABSTRACT 

This conceptual paper aims to clarify the theoretical underpinnings of game based learning (GBL) and learning with 
digital learning games (DLGs). To do so, it analyses learning of game related skills and contents, which occurs constantly 
during playing conventional entertainment games, from three perspectives: learning theory, emotion theory, and 
motivation theory. It is assumed that by an analysis of the processes leading to implicit learning in conventional digital 
games, the underlying principles can be made explicit and subsequently used for designing DLGs effective for curricular 
learning. Theoretical approaches which are used in the analysis include behaviorism, cognitivism, individual and social 
constructivism for the perspective of learning theory. For the perspective of emotion theory, research on learning related 
effects of positive and negative emotion is used, and for the perspective of motivation theory constructs and approaches 
such as self-efficacy, locus of control, interest and self-determination theory are drawn upon. All of these theoretical 
perspectives help to understand how players of conventional entertainment games acquire a wide range of skills and 
contents while playing, and why they do so with immense motivational and emotional involvement. The results of the 
theoretical analysis are subsequently used to deduct criteria and guidelines for designing and applying digital learning 
games as powerful learning environments. Early experiences derived from the checklist’s application are reported. 


KEYWORDS 

Game based learning; digital learning games; theory; learning; emotion; motivation 


1. INTRODUCTION 

Game based learning (GBL) or learning with digital learning games (DLGs) has been one of the most 
discussed and propagated forms of media based learning in recent years. Some programmatic authors (e.g. 
Gee, 2007; Prensky, 2007) are extremely optimistic in regard to the potential benefits of GBL, and there is a 
growing corpus of empirical research on educational uses of DLGs (e.g. Shelton & Wiley, 2007; Tobias & 
Fletcher, 2011). However, little effort has been spent until now in systematically analyzing the theoretical 
underpinnings of learning with digital games (cf. Moreno-Ger et al., 2008). 

This theoretical paper aims at closing this theory gap in research on DLGs. This seems a particularly 
important task, as at the moment there is little but experiential knowledge on what makes a DLG effective for 
learning. Methodologically, therefore we will analyze learning in conventional digital games from the 
theoretical perspectives of learning theory, emotion theory, and motivation theory. Undoubtedly, players of 
conventional digital games are often acquiring a range of skills and contents while playing, and they do so 
with immense motivational and emotional involvement. It is assumed that by an analysis of the processes 
leading to these kinds of implicit learning, the underlying principles can be made explicit and subsequently 
used for designing effective DLGs. Accordingly, we will subsequently deduce criteria and guidelines for the 
design and application of effective DLGs from the previous theoretical analysis. We conclude with an 
outlook on possible applications and further challenges for the theoretical foundation of learning with and in 
digital learning games. 
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2. THE PROMISE OF DIGITAL LEARNING GAMES 

After a period as the “new kid on the block”, digital learning games have been developing into the next “big 
thing” in the area of media based education approaches. Similar to earlier trends such as e-learning, many 
have set enormous expectations in this area. On one hand, these expectations relate to profitability aspects, as 
the market for DLGs is believed to have an enormous potential for growth (see Picot/Zahedan/Ziemer 2008). 
On the other hand, even greater are the expectations of some advocates of digital learning with computers in 
regard to their potential for educational effects. 

Authors such as Gee (2007) or Prensky (2007; 2011) are at the helm of this movement. Their simple, but 
also persuasive argumentation is as follows: computer games that originally were only designed for 
entertainment purposes most often invoke substantial learning processes in players, which vary depending on 
the nature of the game. For example, action and racing games are expected to increase motor and perception 
skills, while design and strategy games will increase forward-planning skills, and adventure games can foster 
complex problem-solving skills. In addition, depending on the background story and scenario of the game, 
users may acquire substantial content knowledge as well. This can occur through the challenging complex 
tasks of a special force commando team in the context of a tactical “Ego-Shooter”, or players may develop 
knowledge of history through historically-based trade or strategy games. According to the proponents of this 
line of argumentation, all these learning processes occur without the learner feeling as if the process is 
difficult, burdensome or uncomfortable. On the contrary, digital games are able to generate an enormous 
amount of motivation which leads to intensive, sustained and emotional engagement with the game contents 
and mechanisms. To some degree, this engagement can extend far beyond the reaches of the game, either 
when users create online communities to exchange information about the game or when they develop their 
own game content in the form of “mods” (modifications). 

Advocates of games such as Gee or Prensky also argue that the undeniable potential of digital games to 
promote unsystematic and implicit learning processes can also be intentionally and directly used to facilitate 
the acquisition of curricular subject matter. They often refer to showcase model projects such as, to name one 
example, the program “Revolution” (www.educationarcade.org/node/357), which is based on a modification 
of the 3D role play “Neverwinter Nights”. Set in the context of the American Revolutionary War, players of 
this game are able to experience social situations firsthand in order to develop historical knowledge about this 
time period (see Forman 2004). In this visually and technically well developed MMORPG (Massively 
Multiplayer Online Role Playing Game) learners are able to take on a variety of roles such as farmer, artisan 
or slave and move around in an authentic Williamsburg setting and interact with other human players as well 
as computer-controlled NPCs (Non Playing Characters). In the context of game episodes (chapters), a story 
thread is generated that enables users to better understand the path to revolution. 

As impressive as milestone projects such as “Revolution” and others may seem, the question remains as 
to whether the principle can truly be applied on a broader scale, such as the proponents of DLGs claim to be 
the case. Setting aside the question of the resources needed to develop such complex learning games, the 
main problem concerns didactic quality. For without a doubt, it is not the games themselves that are effective 
for learning per se. This can easily demonstrated by drawing on negative examples of expensively designed 
DLGs which not necessarily provide an effective learning environment (cf. O’Neil et al. 2005). 


3. LEARNING IN CONVENTIONAL ENTERTAINMENT GAMES 

In order to answer questions regarding the quality of DLGs, we first need to have a better understanding of 
the learning processes that take place while playing the games (cf. Garris, Ahlers & Driskell, 2002). It makes 
sense to first analyze games that are intended not for learning but for entertainment purposes as the idea is to 
take the mechanisms that are effective for learning from conventional digital entertainment games and 
transfer them to the development of digital learning games. In addition, when analyzing computer games 
from the perspective of theory, it is important not only to look at aspects pertaining to teaching and learning 
theory, but also to consider at the motivational and emotional perspectives that play an important role while 
playing these games (see Bartlett et al., 2009). 
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3.1 A Learning and Instruction Theory Perspective 

The first theoretical perspective used in this analysis is the perspective of learning and instruction sciences. 
Here, different theoretical approaches can be used to analyse the mechanisms which foster learning in games. 
The most important are behaviorism, cognitivism, individual and social constructivism (Hense & Mandl 
2009; Woolfolk, 2004). In our context, these do not preclude one another. Instead they should be regarded as 
complementary, since the learning mechanisms proposed by the different theoretical approaches can be 
relevant for different learning goals and outcomes. Furthermore they may be activated to varying degrees in 
different game types and genres. 

Starting with the behaviorist perspective , many games teach new skills and contents via operant 
conditioning with its main principles of positive reinforcement and punishment. Reinforcement in games is 
often realized by successfully mastering a sequence of tasks or levels, by collecting some kind of tokens or 
symbolic currency, or by beating a high score. Punishment, on the other hand, can consist in losing a virtual 
life, failing a level, losing a position in a race or ranking, or by being defeated by either a human or 
computer-controlled opponent. These behaviorist principles are most dominant in action, racing and sport 
games which need a lot of motor and perception skills with little cognitive processing. Here the players are 
continually receiving immediate feedback about the success or failure of their actions. Accordingly, 
behaviorist learning mechanisms can be expected to be most effective in terms of practicing and repeating 
routines, primarily in the areas of perception and motor skills, but they can potentially also be useful for the 
acquisition of factual knowledge. 

From a cognitive perspective, as represented for example by the instructional design approach (e.g. 
Reigeluth, 1983), there are many digital games that can mainly be viewed as problem-solving activities and 
accordingly can train learners’ problem solving skills in different content domains. This generally occurs 
when the players use the information that is embedded either within the game context or the game scenario to 
more or less solve complex cognitive problems. Games that operate on this principle contain a strong 
narrative component and players often have to decide between various potential solutions or alternative 
paths. Adventure and role playing games are classical applications for these principles. In addition to helping 
players build problem-solving skills, these kinds of games can also be used to foster knowledge acquisition 
and increase comprehension. This is accomplished by providing information within the narrative of the game, 
which needs to be applied to the solution of a given problem. 

From a individual-constructivist perspective (e.g. Brown, Collins & Duguid. 1989; Resnick, 1989), games 
may be regarded as providing a rich, authentic and immersing environment for self-directed, discovery-, 
inquiry- or problem-based learning activities. The prerequisite for this are challenging tasks or problems that 
players regard as authentic and relevant, either in relation to the reality of the game that they can to relate to, 
or in relation to their own experiences. Based on such problems, the game then allows the player to analyze 
the situation and to test out a variety of solutions, as well as gain experience with and reflect on a specific 
subject area or phenomenon. Examples of this are strategy and design games, since they are more or less 
based on simulations or aspects from the real-world that serve as a context for the specific activities. 

From the perspective of social constructivism (e.g. Bielaczyc & Collins, 1999), finally, the focus shifts to 
the social and cooperative aspects of computer games. Learning in the context of computer games can here 
be interpreted as the joint construction of socially shared knowledge, as this has been traditionally examined 
through research on learning communities or on collective information processing. The processes that can be 
examined most thoroughly occur in the context of MMORPGs. This is where the players come together as 
teams with clearly defined roles in order to master tasks when the solution requires a high degree of common 
planning and coordinated effort. The players communicate and cooperate with each other not only in the 
context of the game, but also often in community elements such as online forums, chats or instant messaging 
which allows players to coordinate and exchange ideas. 

3.2 An Emotion Theory Perspective 

The influence of emotions on the learning process have often been neglected to date in educational research 
(see Astleitner, 2000). Especially in the context of learning in computer games, it is important that emotions 
be taken into consideration as well. Even if the research to date has been relatively sparse, it can be said with 
a degree of certainty that positive emotions such as joy or satisfaction generally have a positive influence on 


21 



ISBN: 978-989-8533-12-8 © 2012 IADIS 


effective learning (Pekrun 1992). With respect to negative emotions, it is important to distinguish between 
deactivating negative emotions such as boredom or hopelessness and activating negative emotions such as 
fear or anger. While it can be assumed that deactivating emotions generally do not support learning 
processes, the influence of activating negative emotions is more complex. If these are present in the right 
amount, they can have an activating effect, but if they are excessive, they can have a blocking effect (see 
Rheinberg 1999). Even when there is the right amount of an activating negative emotion, it is wise to use 
caution because the motivational effect of negative emotions such as fear or anger is extrinsic and may 
actually detract from the actual subject matter and learning process. 

Fun and joy are the two things that first come to mind when examining individual emotions more closely 
in relation to computer games. If one tries to identify exactly what makes a player experience fun and joy, 
you will hear many different answers (see Choi et al., 1999). Reasons may include aesthetics such as 
graphics, animation, music and sound effects or aspects of the game’s narrative. In addition, games often 
provide players with the opportunity to immerse into a virtual world or to take on an artificial identity and to 
experience the joy of success and other social aspects of the game. It is also important that the joy of playing 
the game is not diminished by too low or too high a difficulty level, through subjective unfairness, or due to 
usability issues. In addition to fun and joy, there are also other positive emotions such as curiosity, 
satisfaction, and pride that can also be beneficial to learning processes. 

With respect to negative emotions, it goes without saying that computer games aim to minimize 
deactivating emotions such as boredom or hopelessness. Activating negative emotions, on the other hand, are 
often specifically promoted. A certain amount of frustration when the goals of a game cannot be achieved on 
the first try is a pre-requisite to motivate players to try a second time. Fear can also play an important role in 
certain game genres such as ego shooters, especially when it plays a part in horror scenarios. However, this 
also highlights the ambiguities pertaining to negative emotions because there are certain mechanisms that 
would not be suitable to be used for processes intended to promote learning. When considering the use of 
computer games for learning purposes, the basic conclusion is that it makes sense to maximize positive 
emotions and to generally avoid negative emotions. 

When analyzing the design of successful computer games from the perspective of emotion psychology, it 
becomes clear that these games generally succeed when they adhere to the principles discussed above. 
Examples of techniques that can be used for this purpose are state of the art design, an adaptive level of 
difficulty, target-group specific virtual worlds and plots, immersing narratives, and intuitive operation. It is 
also important to note that failing to meet one of these aspects may not prevent a game from being successful. 
This indicates that the different aspects that affect emotion psychology may compensate for one another to a 
certain degree. Therefore the individual and varied preferences of the players play an important role and 
should be given due consideration. 

3.3 A Motivation Theory Perspective 

The final important theoretical perspective for analyzing learning processes in computer games is motivation 
theory. There are a number of approaches can be drawn on to understand why computer games often are so 
attractive and motivating for players. The most relevant are constructs such as achievement motivation, social 
motivation, self-efficacy, interest and flow (Urhane 2008). Of particular interest is the self-determination 
theory of motivation (Deci & Ryan 1993), which integrates certain elements of some of the other approaches 
mentioned. It concentrates on explaining intrinsic motivation which is especially effective for learning 
because it is not fueled by external rewards, but is rather directed at the specific activity itself. In the context 
of learning in computer games, it makes sense to examine this approach more closely. 

Self-determination theory postulates that intrinsic motivation depends on fulfilling three basic 
psychological needs: competence, autonomy, and relatedness. Competence relates to the construct of self- 
efficacy and describes the experience when an individual is in a position to be in control and master a 
situation. There is no doubt that this is one of the most important and most attractive characteristics of well 
designed computer games (cf. Salen & Zimmerman, 2004) since they continuously enable players to 
experience self-efficacy. It is also interesting to note that this often occurs through contexts that users often 
do not have access to in real life, such as driving race cars in a racing game, governing a city in a design 
simulation or fighting dragons in a 3-D role-play, a fact which refers to the role of interest in this context (see 
below). 
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In the context of the self-determination theory, autonomy describes the ability to strive towards one’s own 
goals, interests, and aptitudes free from outside influences. While some computer games have a linear 
structure, most offer certain degrees of freedom in specific aspects. Examples for a high level of autonomy in 
computer games can be found in the aforementioned MMORPGs or in other games adhering to the “open 
world" concept. Their main appeal is that they provide a simulated reality and allow players to develop their 
character and its behavior in the direction of their choosing. In these cases, there is often no concrete goal or 
end to the game. Of course there are limitations to this autonomy through the rules of the simulation and its 
limitations. The game’s designers’ task therefore is to offer enough degrees of freedom and incentives to 
stimulate players’ exploration. 

The third important pre-requisite for motivating behavior postulated by the self-determination theoiy is 
relatedness. This can be defined as the feeling of belonging to a social community, whether it be with like- 
minded individuals, peers or colleagues. In this regard, the social elements that are part of modern multi- 
player games have enormous potential. Even outside of the game itself, this can be observed in the many 
online communities that are formed around popular games. It is also interesting to note that feelings of 
relatedness can also develop with virtual characters. This could be with virtual family members in 
simulations such as the popular “Sims” series, or computer-controlled “Party” members in adventure or 
conflict-oriented games that use the help of film-like interim scenes to breathe life into the individual 
characters. 

Two other important motivational constructs beyond self-determination theory should be mentioned as 
particularly important in regard to games, namely interest and flow. Interest can be defined as the special 
relation between a person and a specific content domain or area of knowledge (Krapp, 2005). In regard to 
games, the motivational potential of interest is relevant, as it highlights the role of game genre and narrative. 
Both are important criteria for a game’s success among different groups of players, and it is important to note 
that players of entertainment games are usually free to follow their specific interest in choosing a game. 

A final construct to be mentioned here is flow (Csikszentmihalyi, 1975). Flow denotes an “optimal state” 
of motivated action, in which a person is fully immersed in a challenging task or activity while being skilled 
enough to master this task or activity. As cognition and affection both are entirely concentrated on the 
activity, flow allows a maximum level of performance. To induce flow, a task or activity has to meet a 
number of conditions: it has to have clear goals, the learner’s subjective skills have to match with the task's 
level of challenge, and immediate and informative feedback has to be provided. As good game design is 
careful to meet these conditions, e.g. by successively and implicitly teaching players the skills needed in a 
game, flow can be considered a potent element of players’ motivation. 


4. QUALITY CRITERIA FOR DIGITAL LEARNING GAMES 

What conclusions can be drawn from our analysis of educational, emotion and motivation theories of 
learning in computer games? If one agrees with the argument that digital learning games can make use of the 
mechanisms that are used in conventional entertainment games in order to support intended learning 
processes (cf. Linehan, Kirrnan, Lawson & Chan, 2011), then it should be possible to use the results of our 
analysis to derive theoretically well supported quality criteria for DLGs. On the basis of these considerations, 
we have developed a list of quality criteria for DLGs (Figure 1). 

In the recent past, we have used this list of criteria in a number of practice related projects, which were 
either concerned with supporting the conceptual design phase of DLGs, with quality analyses of early 
versions of DLGs, or with the formative evaluation of nearly finished games. Some important experiences 
have come from these applications. The most important observation was that the full educational potential of 
computer games, as indicated in our list of criteria, is often used only to a little degree. On the surface, it is 
often immediately apparent that many DLGs cannot keep up technically with commercial games due their 
smaller budgets. However, as already indicated, a simpler design may not necessarily prevent a (learning) 
game from being successful, as can be seen in the growing market of casual games and mobile phone games. 
Far more important than technological inferiority, however, would be inferiority relating to educational 
aspects that can be identified using the criteria list. Three problems seem common to many DLGs. 
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1. Clearly define the learning goals of the game without neglecting the playful elements 

2. Make use of the full spectrum of learning principles used in digital games 

a. Behaviorist principles 

■ provide direct feedback (particularly reinforcement) on learners' actions 

■ give opportunities for exercise and practice 

b. Cognitivistic principles 

■ embed complex problems within the game context 

■ embed information needed to solve the problems within the game context and narrative 

c. Constructivist principles 

• create realistic problems which are authentic and personally relevant to the players 

■ offer different perspectives and contexts for a given content 

■ create a social context for learning 

■ provide instructional support 

■ offer opportunities for learners' own construction processes 

3. Evoke positive emotions 

a. Guarantee that learners have fun, e.g. 

■ provide an attractive game design 

■ maximize usability 

■ avoid frustration and disappointment 

b. Provoke learners' curiosity, e.g. 

■ offer different choices 

■ offer opportunities for exploration 

c. Allow for satisfaction and pride 

■ provide positive feedback for learners' accomplishments 

■ create opportunities for presentation of learners' accomplishments 

■ don't let learners fail (too often) 

4. Evoke and keep up motivation 

a. Foster intrinsic motivation 

• make learning and playing intrinsically attractive 

■ avoid too much focus on extrinsic rewards (score, awards etc.) 

b. Allow for feelings of competence 

■ set goals which are challenging yet realistic given the learners' ability 

■ give learners complete control over their success (reduce influence of chance) 

■ ensure frequent and constant opportunities for feeling competent 

c. Provide autonomy 

■ provide freedom choice, but avoid too muach uncertainty about possible negative consequences 

■ provide freedom of action 

d. Enable social relatedness 

■ provide in-game cooperation with real and/or virtual partners 

■ create game-related communities of learners 

e. Meet learners' interests 

■ tailor game subject, narrative, and genre to learners' interests 

■ offer choices for the different interests of different learners 

f. Enable flow 

■ clearly state learners’ goals at each stage of the game 

■ adapt difficulty level to learners' ability and skills 

■ provide constant, immediate and informative feedback 


Figure 1. Quality criteria for the design, quality analysis, and evaluation of DLGs. 

Note: for the sake of applicability the criteria here are presented in the form of recommendations 

Firstly, it is sometimes the case that unsuitable learning mechanisms are used for the wrong learning goals 
and contents. Behaviorist learning through reinforcement has its place, but more when it is important for 
learners to practice and repeat facts rather than when learners must learn new information or when the goal is 
to reach a more in-depth understanding of the subject matter. So it is important to provide for a close match 
of learning goal and learning mechanism in each specific case. 

Secondly, it is often the case that the wide-range of possible cognitive, emotional, and motivational 
mechanisms to promote learning are not utilized and combined in meaningful ways. Instead, there is often a 
one-sided focus on individual aspects such as attractive design, frequent incentives or a strong narrative 
element. However, a good design doesn’t compensate for a less attractive game or learning mechanisms. 
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Frequent incentives lose their motivating power when they are too easy to achieve. And a strong narrative 
element is only captivating when the players have enough opportunities to interact within the virtual world. 
So care has to be taken in cautiously balancing the spectrum of possible learning mechanisms. 

Thirdly, and herein seems to lie the biggest challenge, it is always important that game play and learning 
are synthesized in a meaningful way. Our experience has shown that some products announced as DLGs are 
in fact mere e-learning programs to which a number of game elements have been added. Although there is a 
game-like aspect to these programs, the actual contents might still be transmitted through slide presentations 
or spoken instructional passages, the difference being that these elements have been more or less cleverly 
embedded within a game context. 


5. CONCLUSION 

Our theoretical analyses demonstrate that digital games in fact have a lot of inherent potential to foster 
learning via a number of theoretically well established cognitive, emotional and motivational mechanisms. At 
the same time it is evident that, given the state of the art of DLGs, many applications still fall short of making 
use of the full range of mechanisms and often only realise the most basic promoting functions, such as 
positive reinforcement. Accordingly, the results of the above theoretical analysis can be used to derive a 
systemized list of criteria and guidelines for designing effective DLGs. 

The educational significance of this paper is twofold. For the practice of designing and applying DLGs in 
educational contexts, it gives guidance on what criteria need to be met to make them effective learning 
environments. For further research, it provides a general framework which can be applied for the empirical 
analysis of learning with DLGs. 

Until today, the fact that computer games can provide influential learning environments had mostly been 
considered in the context of research conducted on the effects of media. In the past, this research focused 
primarily on the effect of depictions of violence and has brought forth evidence how these can have short- 
and long-term effects on the experiences and behaviors of regular players (see Barlett et al., 2009). Despite 
these negative aspects, we do not see any reason that the learning potential of computer games cannot be 
used in a positive sense for productive learning processes. 

The accompanying challenges can be seen in the challenges presented in this article that many DLGs 
have been struggling with to this day. At the core of all of these difficulties is a basic issue relating to the 
hypothesis that the advantages of entertainment computer games can be easily transferred to DLGs. The basic 
problem lies in the fact that learning in computer games is something different to learning with computer 
games. The hypothesis above can therefore only be fulfilled if it is possible to truly synthesize intended 
learning processes with game processes. 

However, we see positive opportunities for the effective use of DLGs when central principles of 
educational psychology are considered, as we have summarized in our criteria list. It would be a mistake to 
rely too heavily upon the learning effectiveness of the medium of the computer game alone. The risk is that 
we would again take a promising approach with a lot of potential to effectively promote learning processes 
and ruin it by deficits in the aspects relating to educational theoiy. This would lead to great disillusionment, 
as has been the case with e-learning before. 
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ABSTRACT 

Creativity can be applied to every domain of knowledge and must be seen as an important competence. Supporting social 
creativity across different domains and disciplines in learning communities is an important part of collaborative process 
in both university education and in the context of large-scale international projects. In this paper, we present the 
experiences from the Virtual Summer School in Second Life as an attempt to support creative communities in a 
systematic manner. The Virtual Summer School acted as a forum for the presentation of innovative approaches, 
developments, and outcomes of research projects in the areas of technology-enhanced learning, serious games, and 
collaborative technologies, facilitating the exchange of ideas between students, researchers, and practitioners. To the end, 
we present lessons learned and implications for using this approach for supporting creative communities, outlining some 
recommendations and directions for future work. 


KEYWORDS 

3D Virtual Worlds, creative communities, educational visualizations, social creativity, collaboration. Second Life 


1. INTRODUCTION 

Establishing and nurturing vibrant and creative learning communities is a complex process (Wenger et al., 
2002). Such communities are seen as highly important in developing and spreading new skills, insight, and 
innovation (Johnson, 2010). The notion of a Community of Interest (Col) incorporates the variety and dyna- 
mism that are typical features of a modern workplace (Fischer et ah, 2007). According to (Fischer, 2005, 
Fischer et ah, 2007), Cols have potential to be more innovative and transforming than a single Community of 
Practice if they can exploit “the symmetry of ignorance” for social creativity. Supporting social creativity 
across different domains and disciplines in learning communities is an important part of collaborative process 
in both university education and in the context of large-scale international projects. 

We argue that three-dimensional Virtual Worlds (3D VWs) can benefit creating and supporting learning 
communities. However, it requires a careful design that incorporates various activities and exploits ad- 
vantages of the technology. 3D VW are often seen as a type of social media which are known for community 
support (Jina et ah, 2010) and they have some unique features in addition (Molka-Danielsen, 2011). They 
support synchronous interaction, providing a sense of presence, which is important for the development of 
online communities (Bronack et al., 2008). Many 3D VWs support user-generated content, allowing to leave 
traces of activities, which may become part of the shared repertoire of the community through reification 
(Wenger, 1998). Wide opportunities for interaction and simulating environments make 3D VWs suitable for 
conducting a range of virtual events, including meetings, performances, and role playing (Sant, 2009). 

The above features of 3D VWs extend the possibilities of using boundary objects (Star, 1989) and shared 
artifacts as catalysts of collaboration (Wenger, 1998, Thompson, 2005). Boundary objects are externaliza- 
tions that have meaning across the boundaries of the individual knowledge systems or sub-communities and 
are necessary for overcoming distances in social creativity (Bruner, 1996, Papert and Harel, 1991a). Exam- 
ples of such objects include ‘monuments’ (symbols strengthening identity within the community); ‘instru- 
ments’ (an infrastructure supporting interactive communication), and ‘points of focus’ around which the col- 
laboration is structured (Thompson, 2005). In addition, online communities can benefit from such VW envi- 
ronments being dedicated community spaces (Wenger et al„ 2002). 
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In this paper, we explore alternative approaches to technology-enhanced learning, community building, 
and creativity support. We have chosen the Virtual Summer School as an innovative education form exploit- 
ing the strengths of 3D VW and combining 3D visualizations, role-plays, and virtual events. The Second 
International Summer School on Collaborative Technologies, Serious Games, and Educational Visualizations 
was held in the Virtual Campus of the Norwegian University of Science and Technology (NTNU) in Second 
Life. The school was conducted in conjunction with the Cooperation Technology course at NTNU and orga- 
nized by two research projects supported by the European Union (EU) - TARGET 
(http://www.reachyourtarget.org/) and CoCreat (http://www.cocreat.eu/). 


2. STUDY SETTINGS 

In order to evaluate the effect of the Summer School on learning communities involved, we have conducted 
an exploratory case study. Educational activities of the study were systematically designed using a theoretical 
framework of collaborative creative process (Schneiderman, 2002), as presented below (Table 1). 

Table 1. Creativity phases and Summer School activities 


Creativity phase 

Course activities 

- Collect (searching for material and 

visualizing it) 

- brainstorming the topic to be visualized 

- describing the design in group blogs 

- Relate (consulting with peers) 

- participating in virtual events 

- exploring other constructions 

- Create (trying out solutions, creating 

- collaborative construction 

associations, composing artifacts) 

- accessing building resources 

- Donate (disseminating results) 

- role-play presentations 

- preserving constructions in the Virtual Gallery 


2.1 Collaborative Educational Visualizations and Role-Plays 

Collaborative educational visualizations and role-plays were conducted as part of the Summer School with 37 
students working in 10 groups, 3^4 students in each. The students were required to build an educational 
module representing a major curriculum topic and present it at a joint session by role-playing (Fig. 1). 



Figure 1 . Student visualization project Awareness Lab 


We used pre- and post-questionnaires to identify the previous experience of the participants, their 
expectations of the forthcoming activities, and how the activities conducted matched their expectations. Each 
group was required to keep a blog for sharing and discussing proposals, reflecting and documenting the 
progress, and for the final discussion. In addition, each student was required to keep an individual blog for 
weekly reflection. The final presentations were attended, apart from the students, by representatives from EU 
projects and the general public. The resultant constructions have also been evaluated by students from the 
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College of Education (COE), the University of Hawaii at Manoa (UHM). After the role-play session, each 
group saved its construction in a repository called Virtual Gallery and evaluated the work of two other 
groups. 

2.2 Virtual Events 

Two international events were conducted as part of the Summer School. The first was organized as a seminar 
on EU projects, which included five presentations on relevant topics and a question-and-answer session. The 
objective of this event was to demonstrate to the students how international cooperation can be established 
and supported using modem technologies and to disseminate the results from TARGET, CoCreat, and other 
EU projects, exploring the possibilities for cooperation. The seminar took place in a formal lecture setting, 
with an amphitheater for the public, slide show, and interactive posters (Fig. 2). The event involved about 35 
participants - presenters and the audience from several countries. 



Figure 2. Virtual seminar at NTNU 


The second event was organized as a virtual tour to the virtual campus of COE UHM and augmented with 
a feedback session with an invited expert. The students visited the major highlights of the COE UHM virtual 
campus. They were informed that the goals for the COE virtual campus are creating places for experimental 
teaching and research, socializing and collaboration, outreach, culture, and place for entrepreneurship. The 
visit was followed up by the return visit of the Hawaiian students. The goal of this exchange has been raising 
awareness of each’ other research projects and seeding creative communities based on the joint interests. 

2.3 Method and Data Collection 

Our approach to using educational visualizations in 3D VW has been developed in several previous studies 
(Fominykh and Prasolova-Fprland, 2012). It is based on constructionism - an educational philosophy which 
implies that learning is more effective through building of personally meaningful artifacts than consuming 
information alone (Papert and Harel, 1991b). Constructionism is related to social constructivism which 
proposes that learners co-construct their understanding together with their peers (Vygotsky, 1978). In 
addition, we applied role-playing, which implies an active behavior in accordance with a specific role 
(Craciun, 2010). 

The data were collected from the direct observation of students’ activities, pre- and post-questionnaires, 
virtual artifacts (chat log and 3D constructions), and user feedback in the form of blogs. For data analysis, we 
use the constant comparative method (Glaser, 1965) that was originally developed for the use in grounded 
theory methodology and is now applied more widely as a method of analysis in qualitative research. 
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3. SUMMARY OF THE STUDY RESULTS 

3.1 Collect Phase 

3.1.1 Brainstorming the Topic to be visualized 

For performing the visualization task of the Virtual Summer School, the students had the option of using both 
Second Life and other tools, both synchronous and asynchronous modes. 

Six groups explicitly stated that the process of their project work was creative. In particular, four groups 
(including some of already mentioned) noted that they had a creative and productive idea generation process: 

- Generally, we are of the opinion that our construction process was somewhat more creative than in real 

life. 

Three of the groups noted that their creativity was not affected by the technology as they were 
brainstorming the constructions before starting to work in Second Life and designing on paper: 

- In the beginning, we spent time brainstorming about our project, at this point we ignored any technical 
limitations and decided that we would adapt our idea to these limitations when we started to build. 

3.1.2 Describing Construction Design in Group Blogs 

The students were required to describe the design of the constructions in their group blogs to allow the ideas 
found during the brainstorming to crystalize. Reflecting on this task, they acknowledged its usefulness: 

- Exploring and visualizing the topic textually through blogging had the advantage of allowing a more 
detailed description of the topic and about the functionality of the application. 

Blogging technology was found useful at this stage of the project work by many groups. The students 
mentioned advantages of the technology 17 times, but the disadvantages only five times. Blogging was found 
easy accessible and simple. At the same time, it has low interactivity and weak support for synchronous 
activities, which however, was found to be positive by some of the groups: 

- Another upside is that your mental work will not be disrupted. That might be the number one advantage 
of avoiding instant communication. Disruptive communication may ruin your creative work when you focus 
on intensive thinking. 

3.2 Relate Phase 

3.2.1 Participating in Virtual Events 

After the first virtual event , the students were asked to provide feedbacks to the seminar in their individual 
blogs, identifying both positive and negative aspects. Among the positive aspects, the following themes were 
mentioned most frequently (with the number of students discussing them). 

• Geographical independence of the virtual meetings, allowing the attendance of participants from 
different EU projects and countries (15) 

• The novelty and excitement when facing the technology and learning approaches “different from the 
normal kind of lectures” (5) 

• The comfort of use both for the lecturer and the audience, including low threshold for asking 
questions and the flexibility of giving a talk from own office (8) 

- The main advantage is that you can have lectures with both speakers and audience from all over the 
world. [...] Also, comments and discussions with people from around the world might be completely different 
than what would result from an audience with just Norwegians. 

Among the negative aspects, the following items were mentioned most frequently (with the number of 
students discussing them). 

• Technical problems, especially with the sound, diminishing the overall educational experience (15) 

• Attention distractions both inside (“unusual surroundings”) and outside the virtual environment (e.g. 
accessing social tools) and therefore difficulties with concentrating on the content (6) 

- May be harder to keep focus during the presentation. Easier for the mind to slip when you’re at the 
computer. 
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The analysis of the feedback from the second virtual event showed the different types of learning that 
occurred during the virtual tour. We identified eleven major themes. Those related to creativity and 
community support are (with the number of students discussing them) - campus atmosphere (11), campus 
infrastructure (TO), Hawaiian culture (11), sense of place and immersion (18), and places for informal 
learning (11). It was evident that the majority of the students felt an immersive Hawaiian sense of place. 
However, some students were not convinced by the immersive qualities of the environment: 

- 1 did not feel ‘transported’ to Hawaii as the whole concept of a 3d- simulation does not appeal very 
strongly to me, and I usually draw a very clear distinction between real life and a virtual imitation. 

In the general feedback to both events, the students discussed the possibilities of 3D VWs for 
international collaboration and discussion, communication, promotion, corporate training, and emergency 
simulations: 

- There might be some merit in using 3D virtual environments in creating communities across bounda- 
ries. [...] we want to mention the potential of events; one-time happenings where one is able to gather 
around a common interest at a specified point in time and experience it together with other attendees. 

However, the community building was understood as a long-term process that requires time: 

- We could not really develop a bigger community based on our virtual events, because there was only 
very little time to communicate informally with other participants, but nevertheless they are vital for develop- 
ing a community. 

3.2.2 Exploring Other Constructions 

We explored in what way the students were inspired by other constructions available in the virtual campus, 
including the constructions resulting from the First Virtual Summer School in 2010. The students expressed 
very different opinions from stressing the importance of studying previous students’ constructions to 
mentioning a minor effect of this kind of studying for inexperienced users. Five groups stated that they were 
inspired by the available resources and examples of constructions: 

- [...] the student constructions can stimulate the community development by providing new ideas and 
inspire other people to create their own constructions. 

The students discussed how resources and examples of similar projects available in the Virtual Campus 
affected their creativity. Only one student group stated that their creativity was positively affected by the 
resources and other constructions in the Virtual Campus. The other groups were to different degrees certain 
that their creativity was not affected: 

- We looked at the earlier projects to get a feeling of what is possible of achieving in the given time for 
the project. Of course, our building was a bit inspired of the style of building [...]. 

3.3 Create Phase 

3.3.1 Collaborative Construction 

The students applied different metaphors and design approaches that can be sorted into three main categories. 
They are ‘scenes for their role-plays’ (purposes were too unclear without the presentations), ‘facilities’ 
(workplaces, which visitors could use, games, where they could play, or tools, where a single user could 
learn), and ‘museums’ (exhibition and guided tom' instead of the role-play). 

Half of the groups stated that 3D VW positively affects creativity and supports generation of new ideas: 

- New ideas were often generated by “playing around” with objects without a concrete plan of what we 
wanted to achieve but by combining elements (prims) which we liked into a greater construct. 

At the same time the other groups argued that the technology, being unknown, hinders creativity: 

- It affected our creativity in that manner that neither of us had any experience [...]. So when we were 
supposed to start building, we did not know what was possible, and how to do the things that were possible. 

3.3.2 Accessing Building Resources 

The building resources available in the Virtual Campus were used to a limited degree. Most of the groups did 
not see them contributing to the community support. However, three groups explicitly mentioned these 
resources ease the constructing process: 
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- We discovered elements from other projects and generally around in second life that we wanted to 
incorporate into our [project]. Other things gave us inspiration to try to make ourselves or improve [...]. 

- The amount of previous constructions was small, but it still showed what could be done, and what to 
aim for. Especially the latter might be inhibiting to creativity, as it might not be especially motivating to 
surpass the previously created work [...]. The already available scripts and textures made building cheap, 
although it might lock participant into a narrow thought process [...]. 

3.4 Donate Phase 

3.4.1 Role-play Presentations 

During the final phase of the Summer School, the students were presenting their constructions to other 
participants. In the discussions, all the groups noted advantages of role-playing as a learning activity. The 
most popular of them include efficiency and safety comparing to the real-life training, possibility to have a 
good contact with the audience, and offering experience together with information: 

- 3D role-plays can be useful and sometimes necessary for imitations of real-life situations that can be 
dangerous, or that can happen (but still useful) with some lesser probability. 

The students identified two most serious challenges for such type of activity: not enough realistic 
experience and the amount of effort required to make a play. Half of the groups discussed these challenges: 

- [...] even though we are presenting something based on a role-play we are still in a virtual 
environment. We think that it is not the same having a role-play in virtual environment or in real life. 

Role-playing activity was also found to be an important part of the visualizations. In some cases, they 
clarified the purpose of static constructions. In some other cases, role-playing became the central part of the 
projects, while 3D constructions were serving as a stage. 

3.4.2 Preserving Constructions in the Virtual Gallery 

The students acknowledged the possibilities of 3D VWs for international collaboration, virtual visits, and 
knowledge sharing as it was done in the Summer School events. Virtual Campus of NTNU and generally 3D 
VWs were talked about as suitable for supporting communities in the long term. 

First advantages for community, preservation of constructions, community repository, dissemination 

Sharing 3D constructions received a positive feedback. Most of the groups stressed the importance of 
studying previous students’ constructions to have inspiration. Some of the groups stated also that they get 
additional motivation from exhibiting their construction for other people: 

- Sharing and exhibiting constructions in the Virtual Gallery is good because it can help newcomers 
introduce what 3D VWs [...] are capable of, what is possible to do, what types of collaboration are possible. 

However, a number of strong limitations were identified, such as low accessibility, technical problems, 
and that experience is not realistic enough: 

- The “general public ” uses small computers, mobiles and other platforms that don ’t have the power to 
run 3D VWs [...]. That’s more barriers added to the task. 


4. DISCUSSION 

The objective of Virtual Summer School was to explore learning environments by inviting participants into 
practices where knowledge and insight is emergent from the diversity of the contributions. The virtual format 
of the Summer School demonstrated the possibilities of modern educational technologies for working and 
learning. It was a deliberate choice to organize the Summer School and the corresponding environment in 
accordance with the four phases of creative collaborative process by Schneiderman. In the following, we 
briefly discuss how the activities in the different phases contributed to seeding and nurturing creative com- 
munities as well as how the existing Summer School facilities supported these activities. 

• Collect phase: Brainstorming the topic to be visualized and discussing the design in group blogs 
contributed to establishing an initial domain, engaging issues, insights, and practices for learning 
communities. A set of resources in the Summer Schools such as existing student construction, 
tutorials, and joint feedbacks sessions in Second Life as well as feedbacks to the blogs provided 
initial motivation and facilitation for collaboration and brainstorming in blogs and other arenas. 
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• Relate phase: Participating in virtual events and exploring other constructions contributed to 
establishing new connections and multi-membership in learning communities involved. These 
processes were supported in the Virtual Summer School by providing boundary objects to enable 
dialog and collaboration between learners from diverse backgrounds and disciplines (such as 
exhibition booths and slides from different projects) and by supporting a flexible infrastructure, 
enabling both formal and informal meeting and workplaces for members of different creative 
communities. 

• Create phase: Collaborative construction of 3D visualizations contributed to unleashing and 
supporting social creativity in the participating communities during the Create phase, establishing a 
joint practice and trying out different solutions. This process was supported and motivated by the 
possibility of accessing building resources in the Summer School, both student constructions from 
earlier generations and various building tools and facilities. 

• Donate phase: Presenting the 3D constructions with the role-plays contributed to disseminating the 
results from the participants and projects involved and enriching the reflective dialog in the 
communities with innovative expression forms. In addition, the visualized results are available 24/7 
in Second Life as a part of the Virtual Gallery, thus constituting a shared repository of community 
knowledge. These activities have been supported in the Summer School by providing seminars on 
role-playing in a workplace context as well as storage and rettieval facilities for 3D content. 

Based on our experiences, we can outline the following implications for organizing learning processes 
and supporting learning communities in the context of a virtual summer school. These lessons learned can 
serve as guidelines for educators/project consortia in similar situations. 

• 3D visualizations are important for community building and dissemination of educational content, 
supporting exchange of ideas in a virtual workplace as well as enhancing creativity across 
boundaries of different Cols. Therefore, there is a need to explore alternative and innovative ways of 
visualizing, storing, and managing community knowledge. 

• 3D visualizations provide alternative possibilities for teaching and presenting innovative concepts 
and research results in an easy-to-understand way. These possibilities should be further explored. 

• Virtual events are an integral part of the educational process and, therefore, of the Summer School 
organization and planning. We have explored different types of events and corresponding modes of 
learning. In order to facilitate such events and different learning modes, it is necessary to provide 
both social and educational spaces for community building and collaborative creative activities. 

• Role-playing in 3D VWs constitutes a powerful disseminating tool and an integral part of the 
collaborative creative process. Role-plays can also serve as workplace training for students (as 
identified by their feedback). Therefore, a further exploration of the potentials of role-playing and 
serious games for supporting learning at the workplace is recommended. 

• All the mentioned elements, i.e. 3D visualizations, associated role-plays, and virtual events, are 
interconnected, supplementary to each other, and necessary for creative communities support. For 
example, without the role-plays, the results of the creative process, i.e. the 3D visualizations, were 
not fully obvious. The 3D visualizations served as boundary objects and were, therefore, necessary 
to create joint understanding between different Cols. The virtual events, in accordance with 
recommendations from Wenger, created opportunities for generation and exchange of new ideas as 
well as new ties between communities (Wenger et al., 2002). 


5. CONCLUSIONS AND FUTURE WORK 

In this paper, we have presented the experience from the Virtual Summer School in Second Life as an 
attempt to provide a systematized support for creative communities in a multi-cultural, cross-disciplinary 
context. In this way, a virtual summer school could be thought of as a framework or a technique that provides 
support for community building, collaborative creativity, and idea dissemination. Based on the results of our 
study, we presented some implications for using this approach that might be reused in the future. In addition, 
we identified some challenges, both related to the fluid and diverse nature of creative communities and the 
technology, in particular Second Life. Although the latter was chosen for its general popularity and 
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accessibility, the results will be relevant for other social VWs as long as they support collaborative co- 
construction. 

Our future work will focus on addressing the identified challenges. In addition, we will explore further 
the possibilities of 3D virtual worlds for supporting creative communities, both in terms of the virtual envi- 
ronment design and organizational forms for educational and social events. 
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ABSTRACT 

Although school budgets have plummeted due to federal and state funding reductions, adopting Bring Your Own 
Technology (BYOT), may address monetary tightening while simultaneously infusing 21 s1 century learning. 
Implementing BYOT may provide real, rigorous, and relevant learning for the students while posing higher-order 
thinking questions from the teacher. This paper expands Williard’s Pyramid (Ullman, 2011) by proposing five BYOT 
components with descriptions, examples, and recommendations within each. The five components are security, 
stakeholders, policies, professional development, and financial planning. The paper closes with future predictions as well 
as words of caution regarding BYOT initiatives. 


KEYWORDS 
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1. INTRODUCTION 

Throughout the world, the influence of technology has forever changed the familiar framework of schools. 
We were once constructed with the Industrial Revolution in mind, but now, through the implementation of 
2 1 S1 century skills, we are embracing a new paradigm that no one has ever built before. Through fast-paced 
technology integration, we can no longer educate the way we did, but rather provide a new form of education 
that promotes authentic learning. (Jacobs, 2010). To provide students with optimal learning, one must offer a 
school structure where real, rigor, and relevant learning occurs at all times. One way in which to achieve this 
goal is to break the barriers between students/teachers and his/her learning. The initiative. Bring Your Own 
Technology/Device (BYOT or BYOD), provides the platform for which students become active participants 
in learning both in and outside of the classroom setting by enabling employees and students to use their 
personal devices and connect with school/employers network. Such devices people bring include, but are not 
limited to the following: smartphones, tablets. E-readers such as Nook and Kindle, laptops, iPods, and many 
others. This paper addresses the problem and framework, five components to consider with BYOT, summary 
and conclusion. 


2. PROBLEM AND FRAMEWORK 
2.1 Problem 

As financial budgets become tight and restrictive, school districts and employers may not afford a 
contemporary device for every learner and/or employee. As a result, a gap exists between what is taught and 
what needs to be taught in the classroom today. Are we preparing our students to be productive citizens for 
society as it is or as it ought to be? Or are we preparing them for the past? Students seem more motivated and 
interested to learn from their own devices rather than a teacher lecturing on various topics, yet we often 
prohibit the use of personal devices as described in technology policies and student handbooks worldwide. 
Through the use of Bring Your Own Technology (BYOT), teachers and students can change the focus of the 
classroom and become more student-centered. The role of each member of the classroom may also change to 
reach the student’s maximum learning potential by creating a customized education for each type of learner. 
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2.2 Definition of Terms 

Bring Your Own Device (BYOD) (Norris & Soloway, 201 1). 

Bring Your Own Technology (BYOT) (Stanley, 2012). 

Bring Your Own (BYO) (R.Cave, personal communication, April 26, 2012). 

Bring Your Own Computer (BYOC) (BYOT.ME, n.d.) 

Flipped Classroom - switch of instruction by giving a lecture the night before via video and using class time 
to help students with problems/questions or ask higher order thinking skills (Hart, 2012). 

Mobile (Internet) Devices - handheld devices having internet capabilities such as iPads, netbooks. Kindle 
Fires (Stanley, 2012) 

Tablet - mobile computer operated through a touch screen (PCmag.com, n.d.) 

2.3 Previous Work 

Eric Williard, chief technology officer of Community Unit School District in Illinois, has developed 
“Williard's Pyramid" (Ullrnan, 2011) explaining how one can implement BYOT effectively. These 
components (listed in order from most to least important) are: funding and leadership, tech planning and 
support, standards and infrastructure, hardware and software, and actualize. These five components coupled 
with professional development can be helpful when developing one’s own program (Ullrnan, 201 1). 

2.4 Purpose 

The purpose of this paper is to enhance Williard's pyramid by providing groundwork for effective 
implementation of BYOT and project how BYOT will transform the medium of how students learn. One 
needs to consider many components prior to successful implementation. Some of these include security, 
involvement of all stakeholders, school policy, professional development, and a financial plan for 
sustainability. With any missing components, integration of BYOT may be unsuccessful; however, creating a 
seamless connection among all five components will not only strengthen the likelihood of a successful 
implementation, but also help the students reach beyond imaginable heights. Desired pedagogy for academic 
success may be realized by providing a learning environment where no child is left behind. This type of 
customization may enhance student productivity, engagement, equitable participation, and a wide variety of 
instructional resources while providing ongoing student accountability. 


3. FIVE COMPONENTS TO CONSIDER WITH BYOT 

As an expansion of Williard’s Pyramid (Ulllman, 2011), five components are presented here with 
descriptions, examples, and recommendations within each. The five components are security, stakeholders, 
policies, professional development, and financial planning. 

3.1 Security (Component One) 

Common concerns held by most districts are how can a school monitor one-on-one devices while not 
jeopardizing school security? How can schools provide necessary firewalls to secure confidential data? 
Security is often considered to be the most important focal point when developing a BYOT plan to develop 
necessary safety measures to protect all parties: students, faculty, administrators, and databases filled with 
private information. 

Whether in school or at work, people are using their personal mobile devices regardless of having a 
formal policy of BYOT (McCaney, 2012); therefore, security is at-risk and is an essential part of this process. 
Security issues include infrastructure, bandwidth, wireless networks, access points, and many others. Both 
administrators and members of the technology department need to converse and discuss options regarding 
parameters and privileges for all members to ensure maximum security while maintaining a world-class 
education. In addition, options such as setting up a separate wireless network for students are a viable 
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alternative to maintain security on important information while not compromising teachers or administrators 
accessibility. 

3.1.1 Security Examples 

New Canaan High School in Connecticut (Ullman, 2011) has curbed a security problem into a working 
solution by asking students to register their Apple devices through a MAC address 
(https://sites.google.eom/a/ncps-kl2.org/macaddress) to monitor each individual device carefully. Due to 
budget cuts, Alvarado Independent School (Ullman, 2011) in Texas, has turned to BYOT and has found a 
possible solution to verify the device and user by providing a screen that flashes and asks every user to agree 
to connect to the network similar to using one at a hotel (Ulrnan, 2011). At this time, all devices will be 
updated with antivirus tools and become secure for the network. Although security is at the forefront of any 
new technology integration, parallel efforts often include ill-willed attempts to compromise the systems. 

3.1.2 Security Recommendations 

To unravel the best security, we must provide students the opportunity to safely and securely use devices and 
provide a means for which to store content. Some suggestions for alternate means of security might include 
an additional security code for e-mail, using iCloud or web-based programs such as Dropbox/Evernote, 
google docs, and/or other free web 2.0 tools. Providing these alternatives for student storage will not impact 
the overall security of the school district, but on the contrary, will increase collaboration and communication 
amongst users. 

3.2 Stakeholders need to be vested (Component Two) 

Creating a partnership among all stakeholders may enable “buy-in” and promote success on any initiative. If 
all members see the value of integration, the success may be astounding and all parties involved will benefit 
exponentially. There are several components to effective partnership that extend beyond the walls of the 
classroom. Stakeholders such as superintendents, principals, supervisors, curriculum developers, parents, 
technology departments, and members of the community, must be well-informed and see the importance of 
such integration because it may affect everyone beyond the classroom environment. 

No longer will education be defined by student/teacher, but rather a collaborative and cooperative effort 
by all. The classroom school day will no longer be seven to two, but rather 24/7/365. Through shared 
partnership, all stakeholders may benefit and see the astounding learning we will all gain from such 
implementation. Partnering with technology provides the premise for which maximum learning takes place 
and all stakeholders contribute to the success of such a partnership (Prensky, 2010). Stakeholder buy-in 
enables observation of how BYOT will transform the paradigm of education where learning will be 
continuous, and the school will not be the sole proprietor of information transfer. 

3.2.1 Partnering Examples of stakeholders 

Partnering is a key component to any integration, but also yields success in the classroom environment. 
Creating a partnership between students and teachers is essential for maximum learning to occur on both 
platforms. Both participants shall contribute to the classroom differently. Teachers will develop and design a 
real and rigorous learning environment that provokes the student's passions while also providing the ability 
to integrate technology to enhance learning and create products that represents transference of knowledge. A 
strong partnership between the student and teacher can foster great discovery and provide the structure for 
lifelong learning. Bring Your Own Technology (BYOT) allows this type of learning because it eliminates the 
disconnect felt between what students want to learn and what they are expected to learn. In addition, it 
provides the student with the opportunity to build one’s knowledge and skill set while providing growth in 
and outside the classroom setting. 

The purpose of implementing BYOT at Forsyth (Georgia) and Williamson (Tennessee) County Schools is 
to provide student access to technology to impart student accountability, collaboration, and immediate 
feedback (Stanley, 2012). This initiative provides the student with necessary tools for research and affords 
students to reach their individual potential (Giordano, 2011). Students will be given opportunities to see the 
connections between their lives and learning and will no longer be grouped by age, but rather on ability and 
personal interests. 
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3.2.2 Partnering with Stakeholders Recommendations 

Having a shared, common goal is essential to promote and implement BYOT in one’s school district. To 
ensure optimal success, each member’s contributions may directly affect the group as a whole. Working 
together will not only provide all to excel, but most importantly, enable the student to see the connection 
between school and real life. Students will no longer ask, “why do I need to know this,” but rather say, “I 
need to know this because it will affect me today, tomorrow, and always.” Partnerships provide pathways for 
positive communication to effectively carry out an objective while providing astute knowledge and 
observations that might not be available if working solo. In addition, global collaboration provides growth 
and enhances divert thinking to create innovative solutions to existing problems and unforeseen quandaries in 
the future. 

3.3 Policies (Component Three) 

Permitting the use of personal electronic devices in a school district also influences Acceptable Use Policy 
(AUP) and student handbooks (Harris, 2012). Prior to implementing these changes, one must inform 
students, parents, and staff of required usage protocols (Harris, 2012; Sucre, 2012). Since students will 
experience increased autonomy, they must be taught correct procedures of effective usage of devices to 
augment learning and eliminate improper usage. The school district is responsible for providing a safe, 
learning environment for all students, therefore, protecting against inappropriateness on the internet as 
defined by the Children’s Internet Protection Act also referred to as CIPA (Quillen, 2011). While integrating 
BYOT, the school district must provide a safe, appropriate, and secure network to ensure safety for all pupils. 

3.3.1 Policy Example of Liability 

Policy is very important regarding BYOT and must include financial liability of personal devices (Harris, 
2012). Since these devices are based on the “student paid modules,” (devices that the students pay for, not the 
school district) who will pay if it gets broken, lost, and/or stolen at school? Who will repair the device if 
malfunctioning - carrier, insurance, and/or school? Will the school confiscate the device if the AUP policy is 
broken? 

What alternative means will the district provide if the student (family) cannot afford such a device to 
ensure equitable learning? What happens if a student accesses inappropriate material from a 3G or 4G 
network at school - who is at fault? (Quillen, 2011). When writing policy it is important to address all of 
these concerns to avoid future problems and/or litigation. According to a BYOT Use Contract (p. 176), 
Fayette County Schools (Georgia) has adopted a policy regarding no school liability of personal devices and 
a breach of policy if accessing from a 3G or 4G network. 

3.3.2 Policy Recommendations 

Since BYOT is a new concept, research is limited regarding effective implementation and/or policy; 
therefore, many districts are piloting programs to iron out any problems. Many suggest that prior to a full 
implementation of such initiative, a pilot program be developed with the oldest aged students on a “trial and 
error” basis. This small scale experimentation may supply the school district with valuable information to 
amend existing policy and/or procedure prior to full engagement by all. 

3.4 Professional Development (Component Four) 

Ideally the classroom no longer takes the form of teacher-centered, but rather puts the attention in the hands 
of the student. Strong emphasis is now being placed on various types of learning such as: student-centered, 
problem-based, project-based, case-based, inquiry-based, active learning, constructivism, and learn by doing 
(Prensky. 2010). A shift in focus has taken place from teacher to student, yet the teacher’s role is even more 
vital now than ever. The educator must provide a strong foundation for the students to acquire knowledge and 
skills, but the procedure in which one will learn may drastically change. Since information is easily 
accessible, students no longer need a teacher to pose questions that are available in google, but rather present 
higher-order thinking questions that promote critical thinking, analysis, collaboration, transfer, application, 
and many others. As a result, providing professional development for all educators will not only augment 
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their success, but may provide a greater breadth of knowledge in a rich, technology-based environment where 
learning takes place. 

3.4.1 Professional Development Examples 

Professional development allows all professionals to develop, apply, and hone necessary skills to increase 
learning outcomes. Since technology is becoming more prevalent in schools today, teachers will expand their 
knowledge and skill sets to augment student achievement in the classroom. If school districts provide 
additional educational support in preparing teachers with effective technology integration, our students will 
be best prepared for the classroom, but most importantly, for the future workplace. Having a skilled teacher 
implement technology will advance creativity, collaboration, and partnership amongst all (Puente, 2012). A 
way in which to boost teacher effectiveness in the 21 st century classroom is to provide professional 
development in the form of workshops, conferences, webinars, graduate school offerings, and/or peer-to-peer 
sharing of new research and/or technological advancements through the form of informal learning 
environments, blogs, and wikis, etc. Williamson County Schools (Tennessee) will provide extensive summer 
professional development for teachers to have effective strategies for BYOT implementation. (BYOT 
Summer Professional Development, 2012). These strategies include collaboration among peers regarding 
topics such as classroom management, personal response and communication systems. Bloom’s Taxonomy 
(Churches, 2011), and useful web 2.0 tools. 

3.4.2 Professional Development Recommendation 

The purpose of professional development is to provide personal growth by impacting one’s knowledge, 
expertise, viewpoint, and awareness. Offering appropriate and relevant professional development for teachers 
may contribute to effective BYOT to implement new methodology. Integration of flipped (Hart, 2012) 
classrooms, student response systems (Puente, 2012), and global connection may make learning an integral 
part of one’s everyday life. Teachers will be able to engage students so school will no longer be seven hours 
of Carnegie units, but rather an opportunity to provide a deep, profound, and tailored learning experience for 
the student. Furthering one’s education provides a great module to students while demonstrating the 
importance of lifelong learning. 

3.5 Financial Plan for Sustainability (Component Five) 

Financial sustainability provides the framework for schools (students) to achieve today and plan for 
tomoirow. Allocating funds is essential because it enables the district to provide necessary means for follow 
through of projects, plans, and integration of technology. Having sustainability in one’s district, provides not 
only strong health, but also the necessary resources to flourish (Jacobs, 2010). To ensure sustainability, one 
must allocate financial resources towards this concept to warrant proper bandwidth, infrastructure, personnel, 
and new technology that has yet to be discovered. We must be well-prepared for the possible challenges that 
come along with BYOT, but also in its ability to revolutionize the classroom as we know it. To guarantee 
success, school districts and business administrators must shift funds from textbooks and other supplies and 
move toward supporting technology initiatives for 21 st century learning (Puente, 2012). 

3.5.1 Financial Plan Example 

With any new technology initiative, such as Flipped Classroom (Hart, 2012), BYOT (Quillen, 2011), and 
others in the imminent future, providing sustainability is a vital component for long-term achievement. After 
implementation of one-to-one devices, school districts must still monitor, maintain, and support the necessary 
infrastructure for perpetual achievement. To make BYOT/BYOD/BYOC effective, one must provide 
financial backing to ensure long-term success. School districts must focus on funds to meet 21 st century 
learning initiatives. 

School districts must prudently select certain technology software/hardware as budgets tighten further and 
technology expands exponentially. Districts would be wise to invest additional time in researching 
software/hardware that is not only trendy, but will produce higher learning outcomes. A tool may have “bells 
and whistles,” yet may not augment the student’s learning. Director of K-12 Technology (West Windsor- 
Plainsboro Regional School District in New Jersey), Rick Cave (R.Cave, personal communication, April 26, 
2012), believes that BYO provides an opportunity for increased student outcomes by using one’s own device. 
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internet, and valuable web 2.0 tools. Maintaining the proper infrastructure and security through financial 
sustainability may enable students to safely access valuable materials and create novel products for the 21st 
century. 

3.5.2 Financial Plan Recommendation 

Allocating funds for technology is no longer considered a luxury, but rather an essential component to the 
overall budget of a school district. The main purpose of BYOT is to not only provide a decrease in overhead 
costs (since purchasing devices has been dropped to a nominal fee), but to provide students with various 
opportunities that otherwise could not be granted in a different setting. Adopting a financial framework for 
sustainability of BYOT and other new technology initiatives is essential so students have a fair and equitable 
education. Technology is growing rapidly and the potential costs to any district may be exorbitant so it is 
important to choose wisely to maximize funds. BYOT provides the ideal scaffold of success while 
minimizing overhead costs. 


4. SUMMARY AND CONCLUSIONS 

The purpose of this paper is to focus on implementation of BYOT in any school district by addressing five 
viable components: security measures; stakeholder involvement; policy creation; professional development, 
and financial planning for sustainability. 

Bring Your Own Technology (BYOT) provides the foundation for changing society into a new form. The 
classroom configuration will look different as mobile integration infiltrates every aspect of one's life, 
therefore, affecting the pedagogy of learning. Learners will have a diverse, interdisciplinary, customized 
global education at the disposal of one’s pocket as they access everything through their mobile devices 
24/7/365. This “pocket-based” learning may revolutionize education and demand greater expectations of 
critical thinking and problem-solving. Education will no longer be static, but will provide the learner with a 
real, rigorous and relevant education. This research depicts how BYOT will revolutionize education and 
create a new prototype of learning both inside and outside of the school setting. 

With this concept and the infiltration of constructivism, students will no longer have their heads on their 
desks due to boredom, but rather be active participants by transferring and applying knowledge in a plethora 
of ways. Some examples include: podcasts, vodcasts, videoconferencing, displaying online portfolios, 
blogging, social networking, epals, and many others. This paper addressed the problem and framework, five 
components to consider with BYOT, and summary and conclusion. 

4.1 Suggestions of what to expect in the Future 

Through the use of new and infinite technological advancements, school is changing rapidly; therefore, our 
design and approach shall also change to meet the needs of the individual student. From the classroom setting 
of desks and chairs to a more collaborative approach of cafe style, brick and mortar perspective may no 
longer be prevalent because the needs have changed drastically. In addition, the teacher’s digital blueprint for 
academic achievement, shall outline a new perspective of innovation, critical thinking, problem-solving, 
transference, and application of knowledge. The intent of the new model is to transform passive learning to 
dynamic discovery of knowledge and skills. We shall begin to unravel universal apps compatible for all 
electronic devices, a classroom truly as wide as the world through global connection, semantic web, and an 
equitable learning environment where all students have the opportunity to experience the same education 
despite, race, sex, gender, age, etc. For individuals who cannot afford such devices/services such as the 
internet, school districts might need to consider allocating a budget for supplying school internet from home 
or the ability to rent/buy devices at a lower cost. With infiltration of online information and accessibility, 
cheating and plagiarism may be increased concerns. 

Education in the 21st century has the capability of transforming education into something unimaginable, 
yet probable. Bring Your Own Device/Technology is the first step making this change necessary, but also 
possible. BYOT provides schools and businesses the ability to decrease overall costs through green initiatives 
and blends technology of the person with school/work. The future may hold new dynamics of learning and 
collaboration and may obliterate schoolhouse practices. In the future, tools may be worn versus carried to 
specific locations (Maes, 2009). BYOT may be responsible for the shift in educational pedagogy for we will 
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not amend the existing educational infrastructure, but create an innovative, never seen before, global house 
where Bloom’s taxonomy will not only occur in textbooks, but be seamlessly integrated throughout all 
learning processes. 

Changing the model of education will infuse the importance of the digital age and place precedence on 
the art and science of learning. Students will need to know how to learn through a wide variety of media, sort 
through what is essential/valid and what is not, and demonstrate digital citizenship. The student’s device has 
exponential possibilities of knowledge, yet the role of an educator is to provide the framework that the digital 
world cannot produce. Profound facilitation, questioning, customization, and feedback will change the setting 
of schools as we know them today while providing an incalculable opportunity for student learning both 
nationally as well as globally. Having access to a myriad of resources enables all learners the opportunities 
for growth and constant reflection aspiring for excellence. BYOT is not just another policy or guideline that 
one needs to follow in education, but is the means by which education will forever be transformed. 

4.2 Limitations of Paper 

BYOT is a new initiative for technology integration and has a paucity of research regarding the effectiveness 
on learning outcomes. BYOT is an innovative approach of joining one’s personal technology with school 
technology; therefore, lacking strong statistical data. In addition to the absence of quantifiable results in 
current BYOT pilots, websites/blogs are popping up with potentially biased treatments of BYOT. For 
example, blogs may appear to be providing information to advance the practitioner, yet a closer examination 
suggests a commercial intent to promote the sale of their products/publications, often based on contributions 
to the site. 
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ABSTRACT 

This study investigates how prior content knowledge and prior exposure to microscope slides on the phases of mitosis 
effect students’ visual search strategies and their ability to differentiate cells that are going through any phases of mitosis. 
Two different sets of microscope slide views were used for this puipose; with high and low colour contrasts and students’ 
ability of recognition were investigated. Study group was consisted of forty undergraduates majoring in science teacher 
education at a major university in Turkey. All participants were considered novice microscope users based on their 
ability. Participants prior content knowledge was assessed with “Mitosis Inventory” and they were grouped based on their 
level of prior content knowledge. High and low level content knowledge groups took part in the study. Based on the 
results of the eye-tracking data; prior content knowledge was found to have an effect on participants’ visual search 
strategies and their recognition time for the slide samples that were in high colour contrast. High prior knowledge group 
was able to recognize and identify the phases of mitosis correctly in a shorter period of time in comparison to low level 
prior knowledge group. However, no difference was found between groups for the low colour contrast slides. 


KEYWORDS 

Visual search, prior content knowledge, prior exposure, biology instruction, eye tracking, colour contrast 


1. INTRODUCTION 

In science education, use of visual materials to support verbal communication is a common practice. This 
way, the success of information transfer increases especially for abstract concepts. Visual representations are 
often used and appreciated in science education including diagrams, photographs and illustrations (Lee 2010) 
and they have a significant use in mainstream K-16 science education. Especially in biology education, 
microscopes and in astronomy education telescopes are vital instruments to provide visual information for 
materials that are not visible to naked eye, such as detailed visual information of cells and chromosomes 
provided by microscopes. Unfortunately, in most cases; students’ ability to interpret visual materials is not 
questioned during teaching. Students’ literacy level is a determinant on interpretation of materials. Novice 
learners are reported to struggle and be less successful in determining the relevancy of information during the 
interpretation of visual materials (Patrick, Carter & Wiebe, 2005). 

With the increasing use of visual representations in science education, visual literacy and visual search 
strategies emerge as the vital skills in recognition and interpretation of visual materials at individual level. 
Visual search tasks are commonly used both as a tool and a method to investigate how attention influences 
visual information processing, namely how the visual system extracts and combines features (Zhuang and 
Papathomas 2011). It has been stated that the factors that increase or decrease efficiency of visual search 
could be categorized into image driven factors (i.e. saliency, single features, colour contrast, abrupt onset 
luminance contrast) and knowledge based factors (i.e. familiarity effect, novelty, experience, automaticity, 
pre -knowledge) (Chun and Jiang 1998). Furthermore, it has been stated that human visual attention and eye 
movements are in a close correlation and analysing eye movements would provide objective measures of 
common attentional processes (Hoffman and Subramaniam, 1995; Frey et al. 2011). Since eye movement 
data can provide information on how users receive and process visual information, general trend in literature 
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has been to identify differences among novice and experts’ performance with field related visual materials 
through eye metrics data. 

Based on the data provided in literature, an experimental study was designed to explore the visual search 
strategies of novice learners in relation to their content knowledge level and colour contrast variations in 
virtual representations. More specifically, this study was designed to explore how undergraduate students’ 
visual search strategies differ based on their prior content knowledge and the microscope slides’ colour 
contrast variations when they were presented the same content information. In this case, the content 
comprised of microscope slides of onion root cells that are going through various stages of mitosis. 

1.1 Visual Search 

Visual search is defined as a paradigm in which a person looks for a target item among a set of distracter 
items (Wolfe, 1994) “to determine, as rapidly as possible, whether a target item is present or absent in a 
display” (Enns and Rensik, 1990, p. 323). This process involves two elements, a targeted item and distracters 
(Leonards et al. 2002). In visual search literature, it has been suggested that there two kinds of visual search 
types. These are “feature search” and “conjunction search” (Wolfe 1994; Old and Fockler 2004; Zhuang and 
Papathomas, 2011). 

In single feature search tasks, the target differs from distracters on the basis of a single feature dimension 
(e.g., colour, shape, size) and reaction time and accuracy to search for the target remain generally unchanged 
even though the number of distracters increases in a feature search task. By contrast, in a conjunction search 
task, target differs from distracters on the basis of a conjunction of two or more feature dimensions (e.g., 
colour and shape) and reaction time increases while accuracy decreases by the increase in the number of 
distracters (Old and Fockler, 2004; Zhuang and Papathomas, 2011). In their Feature Integration Theory, 
Treisman and Gelade (1980) suggested that conjunction search occurs in two stages; the first stage of visual 
search is a parallel mechanism where information about one single feature dimension such as colour, size, or 
orientation can help in the search; each type of feature is represented by a separate feature map in the brain. 
The second stage of the model is a serial mechanism where each item must be considered in turn to 
determine whether it satisfies each required feature. When targets and distracters in visual search differ by an 
easily distinguishable feature such as colour, size, or orientation the reaction times decreases. On the 
contrary, for targets that “are not easily discriminable or differ from distracters by a conjunction of features 
(e.g., a particular combination of colour and shape) display size has a large effect on search times and the 
reaction time slope is steep” (Trick and Enns 1998, p.350). 

Ngyren (1996) found that if the target stimulus is visually conspicuous then search performance could be 
up to 83% faster. One of the dimensions of conspicuity is differences of background colour or luminance and 
stimulus colour or luminance. Colour is an influential factor of graphical display and when used 
appropriately can lead to faster search times; conversely, poor use of colour can lead to decreases in 
performance (Ling and Schaik 2002, p. 224). Schaik and Ling (2001) reports that colour combinations with 
higher level of background and foreground contrast lead to better performance in comparison to lower 
contrast. In addition to colour contrast, increasing luminance contrast is also reported to be influential on 
visual search process and it was found that the mean fixation duration increases clearly with decreasing 
luminance contrast (Ojanpaa and Nasanen, 2003). Since Donk and Theeuwes (2001) have argued that 
luminance changes are necessary for marking to occur. Olds and Fockler (2004) suggested to further study of 
the question whether visual marking could occur with the present equiluminant stimuli. Background 
complexity is also suggested to effect visual information processing process and fixation duration and length 
of saccades (Crosby, Iding and Chin, 2001). Increase in the background complexity will also result in 
increase in information processing time and fixation duration. 

Apart from those image-design driven factors (background contrast, luminance, conspicuity etc.) 
individual factors i.e. prior content knowledge, prior exposure, visual ability, have an effect on visual search 
performance. For example, success in identification and recognition of microscope display is found to be 
depending on integration of general knowledge, visual knowledge, high level recognition and reasoning 
(Pani, Chariker and Fell 2005) and content knowledge is reported to be an important component during 
visual processing. Moreover prior exposure to an object is suggested to improve the recognition dining the 
upcoming encounters. The effect of this prior exposure is called “priming” and stated to improve one’s 
ability to identify an item or an object (Schacter and Buckner 1998), not requiring conscious effort or explicit 
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knowledge (Rristjansson, Wang and Nakayama 2002). Existing research focuses on visual search with the 
purpose of identifying specific patterns to support the learning process. Studies conducted with the 
individuals from different professions shows differences in visual search performance generally based on 
expertise level, prior exposure and visual display design characteristics. Various studies (i.e. Cook, Wiebe 
and Carter, 2008) point out the differences in interpretation of graphics among students with high and low 
level of content knowledge as well as among complex designed displays and well-designed displays. Results 
of such studies show a difference favouring experts including identification of visual images or interpretation 
of those (Krupinsky et al. 2006; Pani, Chariker and Fell, 2005) and also favouring well designed visual 
displays’ compensatory effect (Canham and Hegarty, 2010). Experts in various fields learn to recognize 
specific patterns and make connections to the prior knowledge (Pani, Chariker and Fell, 2005). 

Also eye movement studies investigating performance among various expertise levels support this 
difference, indicating that experts perform visual search with fewer saccadic movements with more accuracy 
in comparison to less trained participants (Krupinski et al. 2006; Leung et al. 2007). In addition, literature 
supports Henderson’s (2003) suggestion that prior knowledge has a significant role on eye metrics such as 
fixation. Canham and Hegarty (2010) reports on a study focusing on eye fixations with visual displays which 
shows that with declarative knowledge subjects focus on more to relevant information rather than irrelevant 
information. Understanding students’ interpretation of microscope slides in relation to their level of content 
knowledge and their expertise level, which is developed through repetition, holds a merit for the future use of 
such visual materials in education. Moreover, understanding the effect of prior knowledge and prior exposure 
with or without a well-designed environment is also important in terms of science education. 

1.2 Research Questions 

Shedding light on how students conduct visual search and interpret microscope slides in relation to their level 
of pre-existing content knowledge can help in the use and development of visual aids to support learning 
process in education. Therefore, this study targets novice individuals who are lack of repetition and expertise 
in interpreting microscope slides and investigates their visual search processes with the following questions; 

1 . Do the eye movements during the visual search task in different background colour contrast differ in 
terms of individual’s prior knowledge level? 

a. Do the eye movements during the visual search task in high colour contrast images differ in terms of 
individual’s prior knowledge level? 

b. Do the eye movements during the visual search task in low colour contrast images differ in terms of 
individual’s prior knowledge level? 

2. Do the eye movements during the visual search task in different background colour contrast differ in 
terms of having prior exposure to microscope view? 

a. Do the eye movements during the visual search task in high colour contrast images differ in terms of 
having prior exposure to microscope view? 

b. Do the eye movements during the visual search task in low colour contrast not images differ in terms 
of having prior exposure to microscope view? 

3. Is the interaction effect between prior knowledge and prior exposure significant? 


2. METHODOLOGY 
2.1 Participants 

A total of ninety-eight, 3 rd and 4 th year undergraduate students took part in the study. All participants were 
pre-service elementary science teachers who completed at least two general biology courses and had the 
basic skills of microscope use. 

Prior content knowledge test on mitosis and its phases was administered to those 98 students to identify 
their level of prior content knowledge. Their scores were ranged from 0 to 10 with a M=5.67 mean score and 
sd=2.46. Participants were grouped based on their prior content knowledge levels on the phases of mitosis as 
high, medium or low level groups. According to their prior content knowledge scores, 40 students were 
appointed to either high or low level groups, 20 students to each group. High and low level groups were 
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formed in regard to prior test scores (+/-) 1 standard deviation. Further data collection was conducted with 
the participation of students who were placed in either low or high level content knowledge groups. For the 
purpose of this study, all participants were novices in terms of their experience of using microscopes. 
Demographic characteristic of the study group is presented in Table 1. 


Table 1. Demographic characteristic of the participants 


Demographic Characteristics 


F 

Percentage (%) 

Gender 

Female 

28 

70% 

Male 

12 

30% 

Grade 

3 

28 

70% 


4 

12 

30% 

Prior Knowledge 

Low 

High 

20 

20 

50% 

50% 

Prior exposure to microscope view 

Yes 

20 

50% 

No 

20 

50% 


2.2 Data Collection and Instruments 

The prior knowledge was measured by a content test consisting of 12 multiple choice questions which was 
developed by a field expert who is also one of the researchers. Two more field experts outside of the study 
were asked to examine the questions for content validity. Questions were revised according to their 
suggestions. The test was administered to ninety-eight pre-service elementary science teachers for item 
analysis and reliability studies. The analyses yielded two questions out of 12 to be extracted from the test 
since the item discrimination indices for these were below 0.3. The item discrimination indices for the 
remaining questions ranged between 0.32 and 0.66. The reliability of the test was assessed with KR-20 
reliability coefficient and it was found to be .717. Each correct answer was scored 1 point and wrong answer 
was scored 0 point. The maximum score of the test was 10 and the minimum was 0. 

2.3 Data Analysis 

A 2x2 within subject design is used for the purpose of this study. Participants were grouped by their level of 
prior content knowledge and their statement on either having a prior exposure to slides showing phases of 
mitotic division with the consideration of priming effect. During this study participants were shown a series 
of images consist of onion root cells. Students’ task was to identify any cell/s that is/are undergoing through a 
phase of mitosis then click on the cell with the cursor and select the correct mitotic phase from multiple 
choices provided on the right side of the screen. During this task; participants were expected to go through 
the following steps. 1- conducting visual search 2- recognizing cell/s that are undergoing mitosis 3- recalling 
pre-existing knowledge and identifying the mitosis phase. 

Based on prior knowledge test scores, 40 students (20 students from low prior knowledge group and 20 
students from high prior knowledge groups) were selected among ninety-eight students. Experimental 
sessions were executed individually for each participant on an eye tracking computer accompanied by a 
researcher. Data collection session lasted from 10 to 15 min. The eye tracker was recalibrated for each 
participant before starting the data collection procedure and at the beginning an instruction was displayed 
including an audio-visual example. The eye track data were stored separately for further analysis. 

2.4 Experimental Setting 

Seven different microscope images of cells were displayed on a computer screen for each question. The 
microscopes slide images were grouped in two sets; high and low contrast. High contrast images were from 
slides that are stained with four different stains and had maroon coloured genetic material on a light blue cell 
background. Four of the seven questions were placed in this first set. Second set included three low contrast 
slides with orange coloured genetic material on a yellow cell background. The first question set was obtained 
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from the internet and the second question set was obtained from microscope slides prepared by a field expert. 
A sample question from each of the question set was presented in Fig. 1 . 




Figure 1. Sample question from different level background colour contrasts 

All of the cell images were transformed to an experiment set by Tobii software on eye tracker. Tobii T120 
Eye tracker was used to record eye movement data by two binocular infra-red cameras integrated within the 
panels of the monitor. The eye tracker was paired with 17” LCD monitor with 1280 X 768 resolutions, 120 
Hz sampling rate and an accuracy of 0.5°. Among the eye movement metrics, fixation duration and time to 
first fixation were obtained from eye tracker for this study. Fixation duration is the amount of the time that a 
particular element of a design is viewed. Time to first fixation provides to measure how long it takes before a 
user recognizes a specific target. Undergoing mitosis phases on the cells were drawn as area of interests. 
Those area of interests were the areas which students required to find and recognize the right mitosis phases. 
Participants’ eye metrics data were recorded during the process and eye movement data on those areas were 
collected to analyse. Figure 2 shows the area of interest samples from each question sets. 



Figure 2. Area of interest samples from different level background colour contrasts questions 


The images of the cells were presented on the left side of the screen while multiple choices responses of 
mitotic phases were placed on the right side as seen in figure 1 . Participants were asked to find the cell that is 
undergoing through any mitotic phase and identify the cell by clicking on it with the curser and then to select 
the name of the mitotic phase among multiple choices presented on the right side of the screen. Participants 
were informed that each question could contain more than one cell vision, so they are allowed to select more 
than one cell and a phase consecutively. Each of the questions was displayed 20 seconds on the screen. If 
participants did not click on the forward button, the next question was displayed automatically after 20 
seconds. 


3. RESULTS 

Statistics regarding dependent variables (fixation duration and time to first fixation) of low level prior 
knowledge-high level prior knowledge and having prior exposure-not having prior exposure in different 
background colours are presented in Table 2. 
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Table 2. Eye movement statistics across prior knowledge and prior exposure groups 





Low level prior knowledge 

High level prior knowledge 




Prior exposure 
(n= 1 0) 

No Prior exposure 
(n=10) 

Prior exposure 
(n=10) 

No Prior exposure 
(n=10) 


Fix.Dur. 

M 

1.46 

1.29 

2.79 

2.81 


Sd 

.806 

.764 

1.27 

1.94 

<D 

J3 

T_FirstFix. 

M 

7.04 

7.52 

4.66 

3.98 

5 

Sd 

5.2 

5.07 

3.61 

4.33 


Fix.Dur. 

M 

.818 

.946 

1.28 

.889 

£ 

Sd 

.728 

.813 

.852 

.697 


T_FirstFix 

M 

4.93 

3.52 

3.84 

5.18 


Sd 

3.24 

2.87 

2.31 

5.22 


Descriptive analyses indicate that for high contrast images, students with higher content knowledge (a) 
fixate on the cells faster in comparison to students with lower content knowledge and (b) have a longer 
fixation period of time on the images. For further analysis, a two-factor MANOVA was conducted to 
examine eye movement differences in different colour contrasted images across prior knowledge and prior 
exposure factors. 

Before the analyses, normality and homogeneity assumptions of covariance matrices were tested. 
Normality assumption was controlled with skewness and kurtosis value. Kurtosis value ranged from -.5 to 1.6 
and skewness value ranged from .83 to 1.38. These results confirms the normality assumption is not violated 
(-10<kurtosis<10, -3<skewness<3) (Kline 2003). Homogeneity of covariance matrices assumption is tested 
with Box M’s test and it was found that (Box’s M= 54.049, F=1.428, p = .062). Since the value is not 
significant at 0.05 level, it was accepted that covariance matrices of groups are homogenous. 

MANOVA analysis was executed to determine whether there were significant differences in time to first 
fixation and fixation duration across prior knowledge and prior exposure in blue background (high contrast) 
and yellow background (low contrast) questions. The main effect of prior knowledge was found significant 
(Wilk’s A = . 677), F( 4, 33) = 3.932, p =.010, ;/ p 2 = .323). However neither the main effect of prior exposure 
to microscope view (Wilk’s A =. 989), F( 4, 33) = 0.91, p = .985, ;/ p 2 = .011) nor the interaction effect of prior 
knowledge and prior exposure to microscope view (Wilk’s A = . 880), F( 4, 33) = 1.129, p =.369, ;/ p 2 = .120) 
were found significant. 

In addition, between subject effects of prior knowledge was also examined. It was found that high level 
prior knowledge group was fixated on the area of interests (time to first fixation) significantly in a shorter 
time than low level prior knowledge group in blue background questions (high contrast); F(l, 36)=4.14, 
p=.049, I/,, 2 = .103. In addition, in the blue background (high contrast) questions, total fixation duration of 
high level prior knowledge group was found to be higher than their low level counterparts F(l, 36)= 12, 183, 
p=.001, ;/ p 2 = .253. However, in yellow background (low contrast) questions, no statistically significant 
difference between prior knowledge level groups was found neither in time to first fixation F(l, 36)=. 064 
p=.801, ;/ p 2 = .002; nor in total fixation duration in the area of interests F(l, 36)=. 714, p=.404, ;/ p 2 = .019. 


4. CONCLUSION 

This study investigated the effect of prior knowledge and prior exposure in different colour contrasted 
microscope images. The images used as questions in the study were designed in two different colour 
contrasts, high contrast; maroon targets on blue background versus low contrast; orange targets on yellow 
background, to examine whether the colour contrast facilitate the recognition process in visual search tasks. 

The findings of this study revealed out that for high colour contrast image group prior knowledge has an 
effect in visual search tasks both in time to first fixation which shows recognition and fixation duration on the 
related areas. This finding is consistent with individual differences principle of Mayer (2001) in the 
multimedia design, which states that prior knowledge has a compensating effect on producing own mental 
models in multimedia learning (Mayer and Moreno 1998). In high colour contrast questions, high level prior 
knowledge group recognized area of interest in a shorter time and fixated on correct areas longer than other 
areas. This finding is also consistent with prior research which reports that combinations of colours that had 
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higher levels of contrast between background and foreground and also higher prior knowledge generally led 
to better performance than combinations of lower contrast and lower prior knowledge (Ling and Schaik 
2002; Theeuwes, Reimann and Mortier 2006; Huang 2008; Zhuang and Papathomas 201 1). 

On the other hand, visual performance of high level prior knowledge group and low level prior 
knowledge group was found to be similar on low contrast images, contrary to the high contrast ones. 
Nevertheless, based on the findings of this study it is difficult to suggest that prior knowledge effect has 
disappeared in low contrast background microscope images. It should be noted that time to first fixation 
occurs in a shorter period of time, and the difference is greater for low level prior knowledge group with no 
prior exposure between high and low contrast pictures (M=3.52, sd=2.87 vs. M=7.52, sd=5.07). Although 
this was not a main research question for this study, it may require an explanation. Remember that time to 
first fixation and fixation duration were the two eye tracking metrics used in this study. As a follow up to 
explore this difference, data were revisited and the length of first fixation duration metric results were also 
computed in order to explore whether these first fixations would be taken as an evidence of visual search or a 
mere scanning process. The length of first fixation durations were shorter in low contrast pictures (M=0.32, 
sd=0,l) than high contrast pictures (M=0.5, sd=0.3). Those studying the low contrast pictures had a quicker 
fixation but less time spent whereas those studying the high contrast pictures had relatively fixated later but 
stayed longer on the areas of interests. It can be speculated that their first fixation could just be interpreted as 
an indication of mere eye scanning on the picture. Since visual search and scanning are different cognitive 
processes, additional data are required to be able to predict if the prior knowledge affect disappeared or not. 
Furthermore, as Henderson and Hollingworth (1998) once stated, fixation duration is an indicator of the 
cognitive complexity. Based on the findings in this study, it can be speculated that low contrast images might 
increase the cognitive complexity for participants. Another interpretation could be the presentation order of 
the pictures. The participants were first presented the high contrast images followed by low contrast images 
so these results might be due to the familiarity effect. Yet, further studies investigating the role of fixation 
duration with counter balanced designs are required to be able to draw more solid conclusions. 

Individual differences (i.e. visual ability, prior knowledge, visual attention, working memory, visual 
perception) are important in understanding the visual materials and also in learning from multimedia 
environments. However, effect of individual differences could be sensitive to the design of the learning 
environment. Although people could benefit from their positive individual characteristics in well-designed 
learning environments, ever so novel or familiar, as found by Barry and Lazarte (1995), increase in the 
complexity of material would diminish the advantage of previous knowledge. Reversely, well designed 
display designs found to be facilitating comprehension of students who has low prior knowledge (Canham 
and Hegarty, 2010). Further research could explore the effects of other individual differences such as gender, 
age, and distinctive socio-cognitive characteristics on visual processes in multimedia environments. 

In conclusion, the role of visual representations cannot be ignored in science education. This study 
investigated visual search process by the measurement of eye movements which is an objective measurement 
of cognitive processes. The results revealed that for novices, who are lack of expertise in interpreting 
microscope images, such as participants of this study, prior knowledge has an effect when high contrast 
images were in question. Findings of this study can be useful for both teachers and instructional designers in 
science education. Instructional designers should consider design principles together with individual 
differences when developing visual displays. Teachers also should consider individuals’ abilities and 
properties of visual materials when choosing instructional materials. 

However, findings are inconclusive for the low colour contrast image group due the lack of qualitative 
measures as verbal protocols or cognitive task analysis and further studies are recommended. In addition, this 
study was limited to 40 students at the undergraduate level. Studies with larges samples would be needed to 
confirm the findings across grades and cultural settings. 
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ABSTRACT 

The purpose of this case study was to measure the impact of using ICT in Individual Learning Programmes of students 
with learning disabilities. Twenty five students and thirteen teachers took part in the research which was based on 
classroom observations. The Kappa coefficient was employed as a measure to statistically quantify the students’ 
acquisition and progress in learning computer and literacy skills through raters’ observations. A high correlation between 
Keyboard, PowerPoint, and literacy skills (writing, reading, and maths) was revealed suggesting the significance and 
impact of keyboard training as an essential component towards achieving educational objectives. Moreover, thematic 
analysis of the teachers’ interviews provided a further dimension for understanding factors influencing the ICT 
integration process. These results further confirmed the positive impact of computer skills training on the students’ 
learning experience which also revealed a strong belief in the right of every child to all learning opportunities, 
particularly ICT regardless of his/her abilities. 


KEYWORDS 
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1. INTRODUCTION 

The formula for the effective integration of any type of Information and Communication Technology (ICT) 
in education is not the technology in itself, but rather the perception of why, how and whether it can 
invigorate one's learning. Several researchers have recently presented new insights into the effect of ICT on 
education but mainly focusing on mainstream environment. Unfavorably, the area of ICT and learning 
difficulties is not yet extensively researched and necessitates more peer-reviewed body of literature in order 
to build in-depth knowledge of the optimum means of using ICT effectively with Special Education Needs 
(SEN)s (Abbot, 2007). 


2. RESEARCH CONTEXT 

The research was conducted in a center specialized in children with SEN education who were not 
mainstreamed due to the severity of their needs and will be referred to throughout the research as ‘Center’ . 
The Center operates in Cairo, Egypt and caters services to Egyptian and international students, 2 years to 21 
years old, with autism and other disorders that impact on their communication, physical, mental or social 
development. Students are grouped according to both age and skill level in ten classes where instruction is 
conducted in Arabic except for two English instruction classes. Instruction is conducted in one-to-one and 
small group sessions, incorporating each student's Individualized Educational Plan (IEP) objectives through 
activities that promote cognitive development, language, motor skills as well as academic skills. A computer 
department was structured four years ago in order to integrate computer sessions into students’ IEPs where 
computer skills are taught on a 1:1 basis at different levels to approximately 37 out of 60 students who 
qualified both physically and mentally for this task. Computers are equipped with regular screens, keyboards 
and mouses as well as external speakers. 
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3. METHODOLOGY 

The nature of the impact of integrating ICTs with the IEPs of students with SEN was the main question that 
this research aimed to answer. In particular, the following questions were processed into the research on 
what type of ICT was employed, how to measure the ICT skills of students with SEN if achieved, and what 
were the factors behind the development of this practice. 

3.1 Participants 

An initial selection of 37 students at the Center identified upon an interview with the senior computer teacher 
who agreed to include all students currently enrolled for computer sessions. However, 25 students only were 
observed and rated for performance because of lour parent refusals, and eight absences during the 
observation period. Due to the distinctive nature of the Center students’ disabilities and broad age range, it 
was not possible to select a specific age range for the study and hence this sample was included as a unique 
heterogeneous group of students. At the sample selection phase of the research, a plausible grouping of the 
students was made according to their formal assessments and also, due to the severity of cases, ability took 
more precedence than age in the students’ assignment to different classes. Therefore, Group A included 
autism diagnosed students with variable severity: N=16 mean age 12;5 (range 4;5- 24; 10) 13 boys and 3 
girls; Group B included Down Syndrome diagnosed students: N=4, mean age 10;9 (range 6;8-14;7) 2 boys 
and 2 girls; Group C included Cerebral Palsy diagnosed students with variable severity: N=3, mean age 12;5 
(range 1 1 ; 1 - 14;9) 2 boys and 1 girl; Group D included one 3 rd chromosome deficiency diagnosed boy age 
6;5, and finally Group E included one moderate developmental delay diagnosed boy age 21;8. Ten class 
teachers, three computer teachers, a Center coordinator, and four parents were invited respondents for the 
interviews. 

3.2 Data Collection Methods 

A pilot interview with the Center coordinator and senior computer teacher showed that computer skills were 
taught with the objective of setting up the Center students to be computer users. Hence, due to the 
unavailability of sufficient records containing information about students’ performance in computer sessions, 
some were partially developed in response to the researchers' request. Students’ background information, 
formal assessment of students’ skills in relation to dealing with different types of ICT, reasons for approval 
or rejection to join specialized computer sessions, task dates and specific measurement of performance 
throughout the period of computer training, progress data, sessions’ observations, and more were examples of 
required and requested data for establishing a clear history and also for comparative objectives over different 
time periods. On the other hand, the available data representations were a comprehensive checklist of 
sequential categorized computer skills used for an informal evaluation of each student’s current computer 
skills before starting sessions in addition to a non-descriptive register of progress in yearly reports. A mixed 
design approach combining the application of both qualitative and quantitative research methods (Johnson & 
Onwuegbuze, 2004; Hanson et al., 2005) was adopted to explore the current practice of integrating computer 
sessions in students’ IEPs and register progress whenever verified and thus demonstrate the impact of 
training students with SEN to be computer users as an ICT intervention. 

3.3 Kappa Measure of Agreement on Learned Skills 

Selection of the quantitative pole in the mixed design aimed at obtaining a quantified measurement of the 
practice investigated. One major preoccupation of quantitative method is measurement as classified by 
Bryrnan (2001) which generates consistent results and provides basis for comparative measurement over 
periods of time as well as among other researchers. To utilize this concept, the Fleiss (1981) Kappa 
Coefficient was selected to measure the agreement of more than two raters on the occurrence of progress in 
learning computer skills for the Center’s students with SEN. This measurement corroborated the occurrence 
of progress as acknowledged by the teachers in the interviews and enhanced the internal validity of the 
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findings. It also served as a baseline for prospective measurements over periods of time to capture 
anticipated developments and/or limitations to this practice. 

3.4 Procedure 

Based on a comprehensive list of the achieved computer skills per student, a rater sheet was developed for 
each student in the sample with the specific skills expected to be observed and rated by the judges. The rater 
sheet divided the skills on the y-axis into various categories: Mouse Control, Keyboard manipulation. Word, 
Paint, PowerPoint, Internet Explorer/email, and academic objectives. The academic objectives were selected 
by computer teachers from the annual multi-departmental IEPs based on their applicability for drilling using 
software programs. On the x-axis, each skill was evaluated upon various attributes: Independent without 
support, independent with verbal support, independent with physical support, not achieved, compliance, and 
motivation. A general performance scale of 1-2-3 was assigned for all attributes except for an ‘X' for not 
achieved where 1=30%, 2=50%, and 3=100% to prevent binary evaluation and add more depth into the 
measurement analysis. Four raters were selected: the senior ICT teacher, the Center coordinator who is also 
a professional SEN tutor, and the researchers. A meeting for all raters to explain the attributes and scale was 
held before the five pilot observations. A cross validation meeting was held after the pilot observations and 
raters commented on the limitation and ambiguity of the performance scale, which was agreed to be changed 
to: 1= 0-30%, 2=30%-70%, and 3=70%-100%. The senior teacher also commented that his pre-knowledge 
of the students' performance interfered with his judgment and that he tended to be tougher, so, the objective 
of rating was re-explained emphasizing the need for fairness and practical accuracy. It was possible to re- 
observe the same students later, which yielded the same results for three raters but with variation towards 
higher rating in the teacher’s sheet. In reflection, post-observation meetings proved to be valuable in 
maintaining the same perspective for all raters which were held consistently until closure of observations. 

Calculating the kappa coefficient (K) developed into a complex matrix due to the addition of 
measurement layers of attributes and performance scales. For this reason, a complex excel worksheet was 
designed to calculate the final coefficient taking into consideration the multiple attributes and several 
measurement layers and also solve the kappa algorithm per skill. To further expand the dimension of 
interpretation and reveal more informative results, a specific software application was designed for two 
objectives: firstly, counter-calculate results as means of cross validation and secondly, explore further 
relations between attributes. The application aimed at: first, the creation of a database for the evaluation 
results of n raters to a number of students based on: a. students’ data (could be grouped into various 

classes/categories)., b. raters/judges data, c. Skills classified into two levels (Main skill and subset of skills 
related to each main skill )An evaluation scheme to represent the grading marked by the rater. Grading 
entailed two levels: (Main level, A,B,C...) and Detailed level (Al, A2, A3...). Secondly the aim was to 
analyse the database content by student and per skill for all students, calculating Kappa coefficients and 
correlations between compliance and motivation during computer sessions. 

The application was designed to retain specific features to ensure replication of evaluation at different 
periods of time for any number of students and raters to monitor pedagogy outcomes and inform practice. 

3.5 Semi Structured Interviews 

Semi-structured interviews were held and taped for re-examination of data and reduction of internal validity 
threats (Robson, 2002). Thirteen teachers, one coordinator, and four parents were interviewed where 
teachers’ interviews consumed approximately forty minutes each and the parents an average of 25 minutes. 
All interviews were held in Arabic since it is the native language for all respondents. Thematic analysis was 
employed as a flexible analytic tool to decipher the flow of data in the semi-structured interviews Braun & 
Clarke (2006). Themes were recurrently reviewed and categorization was refined as deemed crucial Two 
independent readers were called for examining researcher’s classification of two interviews and classification 
of themes upon which there was 80% agreement on the suggested main themes and 100% agreement on the 
assignment of utterances to the different themes as reported in the results and discussion. 
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3.6 Kappa Coefficient Values 

The four raters rated independently the performance of these students during 1 : 1 computer sessions held in 
the Center. A total of 101 computer skills were observed among the 25 students, each according to his/her 
IEP. Selection of skills was initially grouped by program type: mouse control, keyboard manipulation, using 
CD drive. Paint Program, Word program, PowerPoint program. Internet/email and academic objectives. 
However, the type of data found during the rating phase and the outcome of results signified the need for a 
different categorization since not all students displayed achievement through common routes. Therefore, an 
alternate categorization of three major skill divisions was derived as follows: 

1 . Basic computer skills : incorporating mouse control and keyboard manipulation, and using CD drives. 
All 25 students were observed using basic skills since they are the fundamental keys for operating 
computers. 

2. Intermediate computer skills : incorporating Paint, Word, PowerPoint Programs, and Internet/email 
which are considered interactive computer tools and require good basic skills. 

3. Digital Literacy skills : where learning and/or consolidation of classroom academic objectives that 
take place via computer programs (reading/writing/math). 

In the raters’ sheet, five main variables comprising ‘Independent without support’, ‘Independent with 
physical support’, ‘Independent with verbal support', ‘motivation’, ‘compliance’, employed the same rating 
scheme of 1/2/3 where 1= 0-30%, 2=30%-70%, and 3=70%-100% which constituted sub-variables for each 
of the main variables and a sixth variable of ‘not achieved' attribute which required a binary rating of ‘x’ or 
‘no mark'. Raters used the same rating scheme for consistency. Tables 2 through 9 demonstrate the 
calculated K coefficient for all observed skills grouped according to the three new categories. 


4. RESULTS 

According to the senior computer teacher, all skills included for rating and observation were entirely new to 
these students before joining the program. In other words, the rated computer skills value before the 
intervention could be set up to nil at the beginning of training. Achieving a K value for a skill represented the 
students’ ability to perform the task based upon raters’ scores during observation, which in other words, 
signified progress in learning that skill. However, the variant K values represented the degree of agreement 
among raters regarding the level of mastery rather than the skill acquisition. For example, 1.00 indicated 
‘almost perfect agreement' by raters on one specific level of mastery versus 0.39 that indicated ‘fair 
agreement' by raters on one specific level of mastery and accordingly -0.50 indicated ‘poor agreement’ by 
raters on one specific level of mastery rather than unsatisfactory knowledge. 

4.1 Basic Computer Skills 

Since basic skills represented a main threshold for operating a computer, all 25 students were observed and 
raters agreed that these students acquired these skills and could master it at different levels. None of the 
raters marked ‘not achieved' for any of the basic skills but rather showed variable dependency rates where 
some students required either verbal or physical support to accomplish the task. Moreover, independent 
observation did not register a negative impact on the students’ general performance as the students seemed to 
understand the task and act instantaneously upon instruction. However, some students displayed slower 
reaction due to behavioural or disability issues which was compensated by the instructor’s verbal or physical 
support. The K values for both mouse and keyboard skills conveyed 90% and 63% mastery of skills 
respectively. Comparably, as stated in Olson et al.’s (1997) study, the researchers called attention to the 
importance of mouse control training as a basic skill for using the software program prior to starting a 
computer-based intervention which set the participants up for improved performance as computer users. 
Additionally, a high correlation was revealed between keyboard skills and higher skills learning as will be 
shown later in this section, that further corroborates the basic skills value suggested by Olson et al. (1997). 
Interestingly, a teacher during her interview supported this notion mentioning: ‘ ...especially that their mouse 
skills is always better than their hand skills in moving flash cards in a task while working on a table ...’ 
(interview no. 04, lines 12 & 13). 
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4.2 Intermediate Computer Skills 

K values for Paint, CD drive. Word, and PowerPoint programs were calculated. The number of observed 
students for each skill varied according to each student's IEP. 

Paint Program: There was perfect agreement for skills: Reduce and enlarge size of geometric figures. 
Create a group of fruits' images and Save working file but poor for Choosing a colour from tray. 

Using CD Drive: There was perfect agreement for skills: Open CD ROM and Insert disc into drive but not 
for Close CD ROM. 

Word Program: Almost Perfect to moderate agreement (1.0-0.49) for most skills (9) in this category and 
fair agreement for 3 skills (.3) 

PowerPoint Program: K was from 1 to 0.32 for 12 out of 17 skills in this category showing high overall 
agreement. 

The higher K agreement values on both Paint program and CD drive resulted from competent students’ 
performance and small number of students observed (n=4). Students' performance on Word gained high 
agreement among raters with poor agreement on mastery level of only 3 skills out of 12. The PowerPoint K 
values showed total rater agreement on achievement of learning program functions with variant mastery 
levels. Of those, 12 out of 17 skills gained high K values which suggested high mastery of skills. However, 
it should be noted that the remaining 7 skills also obtained rater agreement for achievement but with 
dependant performance on verbal support. None were marked ‘Not achieved' by raters. Internet and email K 
values also followed the same pattern of achieved performance as the preceding programs with variable 
mastery levels and no ‘Not achieved’ markings. Here, 5 out of 8 internet/email skills gained raters’ 
agreement for independent performance yielding high K values while the rest of skills were acquired but 
required verbal support. 

Abbot (2007) stressed the value of graphic symbols in software programs and its impact on facilitating 
the use of the various software functions. In this framework, the software programs employed in this 
research especially Word and PowerPoint highly utilized graphic symbols. It was observed that students 
trained on these programs mostly relied on these symbols, particularly non-readers, for choice/selection of 
tasks that markedly supported them in using the programs to a far extent. Considerably, interviews with all 
class teachers escalated the value of students' learning to use basic software functions to its possible impact 
on their future prospect. For example, teachers expressed their contentment of the students’ progress in 
keyboard skills in conjunction with using the Word program for the objective of typing text. One computer 
teacher described the level of two students (Child07 and Child 14) as ‘...faster than me in some 
tasks ...’ (interview no. 01, line 90) and another class teachers upgraded this opinion to the level of actual 
employment opportunity in a clerical job mainly responsible for typing text in a factory for Childl 1 ‘. . .and I 
employed these computer skills in the factory... ’(interview no. 05, line 72). It is noteworthy that all class 
teachers concurred with this opinion and perceived this stage as a possibility given the right conditions to 
develop it which may be further researched in another context. 

4.3 Literacy Skills 

The observed students utilized their acquired computer skills as a tool for learning and drilling on their class 
academic objectives. The raters’ agreements indirectly involved an evaluation of the students’ academic 
performance regarding these specific tasks, nevertheless, none was marked ‘Not achieved' which signified 
total agreement on the acquisition of skills but with variant mastery levels as per raters’ agreement from 
‘almost perfect’ to ‘poor'. There was almost perfect agreement for the following skills: Re-arrange words to 
form correct sentence. Read passage and choose word opposites Size concept (small/big) choose correct 
image upon verbal prompt, Same/Different concept choose correct images upon a verbal prompt , Select 
correct color from an array of 3 colors upon a verbal prompt and Categorize objects.; Moderate agreement 
(.49) for skill: Choose accurately currency upon verbal prompt and poor agreement for all other literacy skills 
observed. This category may be perceived as a further utilization of the previous acquired skills in the 
preceding two categories of basic and intermediate skills. It entailed the drill and practice aspect of 
technology discussed by Abbot (2007) as using technology to train or rehearse. 
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4.3.1 Correlations across Skills 

At this point, the results pointed confidently at all students’ successful acquisition of basic computer skills 
and the continued progress of some students throughout the following two phases of intermediate skills and 
using computer skills as a learning tool. Further investigation of these results yielded interesting correlation 
results. Various cross-relations among the above data components were examined and thus revealed two 
major correlations between the K values of academic objectives and other programs and another correlation 
between keyboard skills and PowerPoint skills. 


Table 1. Correlation between agreement values of academic skills and other programs. 


K Values 

Correlation Coefficient 

Correlation Classification 

Academic Skills 

Dependent variable (reference skill for correlations) 

PowerPoint 

0.62 

High 

Keyboard 

0.51 

Average 

Paint 

0.32 

Slight 

Word 

0.17 

Low 

Mouse 

0.17 

Low 


As shown in table 1, correlations between each program skills and academic skills provided interesting 
results. The two highest correlations with acquiring academic skills were PowerPoint and Keyboard skills 
respectively. To further establish the relationship among the three highest correlation values presented in 
table 1, another investigation was calculated to determine the magnitude of basic skills category training, 
mouse control and keyboard manipulation, in relation to students' performance in other programs. Table 2 
presented the results of a correlation matrix between keyboard skills and other programs as shown below: 


Table 2. Correlation between keyboard skills and other programs 


K Values 

Correlation 

Coefficient 

Correlation Classification 

Keyboard Skills 

Dependent variable (reference skill for correlations) 

PowerPoint 

0.75 

Very high 

Academic skills 

0.51 

High 

Paint 

1 

Almost perfect 

Mouse 

0.23 

Slight 

Word 

0.15 

Low 


Table 2 shows the highest correlation relationship between a trio of keyboard skills, PowerPoint and 
Academic skills suggesting a strong relationship in the acquisition of skills among these programs. The 
preceding quantitative analysis and discussion of the students’ performance corroborated by the teachers’ 
judgments reasonably illustrated a positive impact on their learning as a result of integrating computer skills 
training into their IEPs, which provided an answer to the main research question. As demonstrated in the 
above tables, each of the 25 students was observed to master a set of skills according to his/her ability and 
training duration. The demonstrated progress was measured on a base of zero computer knowledge of the 
rated skills as formerly mentioned. In fact, it is important to note that other interesting correlations between 
motivation and compliance of students during computer sessions emerged in the course of data analysis 
which may be further researched and reported in another context. 


5. CONCLUSIONS 

The acquisition of computer skills was evidenced for the 25 participant students by the quantitative measure 
of Kappa in a mixed design methodology. The results further unfolded a range of acquired skills at mastery 
levels shown in the range of K values. Furthermore, data analysis revealed a possible categorization of skills 
where the population of students were distributed according to their level of achievement in learning 
computer skills. These categories began with a basic skills level, intermediate, and concluded with literacy 
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skills learning category. Subsequently, correlations across learned skills were computed and revealed a high 
correlation relationship between Keyboard, PowerPoint, and academic skills, suggesting the significance and 
impact of keyboard training as an essential component for further learning of academic objectives. Results 
further demonstrated the impact of computer skills training on learning literacy skills. Learning and/or 
drilling of academic objectives such as basic reading, writing, and math was evidenced for students who were 
able to advance through the basic and intermediate skills categories. This finding further confirmed the 
significant impact of ICT as part of the students with SEN learning experience and a possible additional 
route for overcoming fundamental learning barriers. Additional correlations between other variables such as 
motivation and compliance were also captured during the observation and rating phases, which will be 
reported in another study. 

To corroborate the above findings, teachers and parents opinions were obtained to capture their insiders’ 
perspectives incorporating the researchers’ observations. They further confirmed the positive impact of 
computer skills training on the students’ learning experience. Findings highlighted the need for more 
research on the value of current and potential applications of ICT for students with SEN to position them 
among the major beneficiaries of the technology revolution. Within this framework, it may be suggested to 
replicate this study in the same context for comparative value of ICT impact issues and also in different 
contexts using the same as well as other types of ICT. Moreover, it may be beneficial to suggest the need for 
incorporating this report’s findings with pedagogy aspects and parental perspectives in future research. 
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ABSTRACT 

Asking why is an important foundation of inquiry and fundamental to the development of reasoning skills and learning. 
Despite this, and despite the relentless and often disruptive nature of innovations in information and communications 
technology (ICT), sophisticated tools that directly support this basic act of learning appear to be undeveloped, not yet 
recognized, or in the very early stages of development. Why is this so? To this question, there is no single satisfactory 
answer; instead, numerous plausible explanations and related questions arise. After learning something, however, 
explaining why can be revealing of a person’s understanding (or lack of it). What then differentiates explanation from 
information; and, explanatory from descriptive content? What ICT scaffolding might support inquiry instigated by why- 
questioning! What is the role of reflective practice in inquiry-based learning? These and other questions have emerged 
from this investigation and underscore that w/n’-questions often propagate further questions and are a catalyst for 
cognitive engagement and dialogue. This paper reports on a multi-disciplinary, theoretical investigation with a view to 
informing the broad discourse on e-learning by identifying a specific frontier for design and development of e-learning 
tools. Probing why reveals that versatile and ambiguous semantics present the core challenge - asking, learning, knowing, 
understanding, and explaining why. 
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1. INTRODUCTION 

In introducing any topic of investigation it is usually helpful to understand why it is presented and what its 
key drivers are; a listener or reader often finds it helpful to understand the context of an investigation in order 
to make some initial sense prior to embarking on giving it further attention. Such context can also be 
described in terms of motivation, purpose, rationale, and/or justification for the work - or as “advance 
organizers” (Ausubel, 1960). Perspectives that emerge from responding to questions help to explicate some 
context - e.g., (i) why is this paper submitted to CELDA 2012? (ii) what is the central argument of this 
paper? Providing perspective of this kind can serve as a ttigger for cognitive engagement and doing so - in 
the form of a well-consttucted abstract - is an established academic convention. Roots of influence for this 
practice stretch back to the time of Aristotle, when logos was elaborated as well-formed argument within 
reason and, as such, one of three modes of persuasion - the others being ethos and pathos. 

Thus, motivation for this paper emerges from consideration of future prospects for e-learning activities 
that probe the why dimension, or inquiry that involves why - asking, learning, understanding, knowing, 
reflecting upon, and explaining why. In order to develop an overarching narrative a number of interrelated 
topics are discussed: the evolution of e-learning; the role of questioning while learning; descriptive versus 
explanatory content; inquiry-based learning; scaffolding using information and communications technology 
(ICT); and, future prospects for ICT tools that support and promote w/zy-questioning. 

As a consequence of extensive academic literature on the subject it is assumed that asking why is an 
important foundation of inquiry and fundamental to the development of reasoning skills and learning. But, in 
direct conUast, sophisticated ICT tools that directly support this basic act of learning appear to be either 
undeveloped or, at best, in the very early stages of development. Why is this so? On the one hand, this paper 
suggests there are a number of very good reasons; on the other, it is focused on the implications of 
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developing better tools that support asking why and understanding why in the specific context of e-learning. 
It summarises relevant research with a view to informing the design of ICT tools that might stimulate deep 
learning and cognitive engagement. In supporting a clear rationale for the investigation the following real- 
world scenarios describe common, contemporary situations in e-learning involving concepts of why: 

1.1 Scenario - University Student 

Sarah is a university student majoring in international relations and history. She has opted to do much of her 
studies online because it provides her with the flexibility to take on some part-time work. The university has 
invested considerable funds into preparing appropriate content and assessment tasks for subjects offered in 
online mode; it has also implemented a standard single -platform policy and installed BlackBoard, a Learning 
Management System that helps structure learning content and contain interactions between staff and students. 
Sarah uses Google to search for additional resources for an essay on the conflict in the Middle East. While 
she finds numerous resources it is challenging for her to understand the causes of this conflict or what the 
appropriate actions might be for it to be resolved. The course resources seem well-structured but she is 
required to investigate sources beyond the prescribed texts. If she searches Google with ‘why’ questions she 
feels very dissatisfied with the quality of the results. Likewise, when searching the library catalogue she is 
overwhelmed by the volume of resources and is not confident in making a judgement about why this conflict 
seems so deeply problematic because she finds so many plausible, yet contradictory, and politicized 
explanations. Even though she has access to a number of ‘social software’ applications that enable her to 
interact and share resources easily with others who might be investigating the same topic she feels like there 
is something lacking in the online tools available. She feels that she needs assistance in discerning fact from 
political rhetoric and some other way of navigating and evaluating the large amount of content on this topic. 
She wants to understand the key issues at the heart of the conflict. 

1.2 Scenario - High School Teacher 

Dave is an art teacher at a high school with two decades of experience. The school has a reputation for 
adopting ICT into the curriculum wherever possible; however, art has been the last subject to embrace ICT. 
This is partly due to the fact that Dave feels more comfortable using traditional media. The school is now 
urging him to make the shift. In moving his content into an electronic mode Dave discovers that he has to 
anticipate many of the questions that students typically ask when in the classroom (‘but why do we have to 
study Matisse, sir?’: ‘why is some abstract art seen as having great merit while some doesn’t?’", ‘why are 
there different versions of what constitute primary colours?’). Because of his experience he knows that the 
students need good answers to such questions so that they can be motivated to leant. He thinks that he may 
need to create a bank of such questions together with suitable answers but hesitates because he knows that 
when students ask questions a longer conversation often proceeds. He is unsure of the best way to make such 
information available so chooses just to make it explicit in the introductory text to each task described in the 
online version of his course. But he remains sceptical that anticipating such questions in a ‘canned’ way will 
be as motivating for the students as being able to respond in real-time. He would prefer to foreground student 
questioning and make it stimulating and interactive, rather than content that students read. 

1.3 Scenario - Instructional Designer 

Thor is an instructional designer for a publishing company that specialises in de-mystifying science. The 
publisher has already had commercial success in preparing online materials that mimic the successful 
television series in Australia during the 1960s, ‘Why is it so?’ Thor has been asked to assist in developing 
innovative pathways to scientific content that will stimulate students to think and ask ‘why’, to motivate their 
curiosity and that leads them into understanding scientific inquiry. He is not quite sure how to proceed and is 
suspicious of Q&A approaches because providing answers can often close down inquisitiveness; he knows 
that powerful search engines like Google can deliver responses to search queries but will also limit the 
student to searching, not questioning", he also aware that none of the natural language search engines he 
knows of seem to do a very good job with responding to why questions. How is he to proceed? 
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1.4 Scenario - Teacher Librarian 

Lisa is a teacher-librarian. She has access to a range of repositories of high quality, digital learning content. 
Most of this content is described using Dublin Core metadata (i.e., information such as the author, title, 
keywords, and abstract) and some of it is described by IEEE Learning Object Metadata (i.e., information 
similar to Dublin Core metadata but also includes information about the educational level associated with the 
content and duration of the resource). Some resources also have metadata that describes associated learning 
objectives and competence level required in order to interact effectively with the resource. Lisa has found 
that many of the teachers she supports also want to know why a particular resource might be more suitable 
than another for a particular learning activity or goal. Lisa has found that rating systems and user-generated 
tags and ‘folksonomies’ are sometimes helpful in this regard but is frustrated that not all the repositories 
support such services. She wonders whether there might be a better approach. 


2. E-LEARNING EVOLVES 

2.1 Terminology and Scope 

Theory and practice of any domain of human activity are constantly evolving and mutually informing. But 
while both philosophers and practitioners have discussed matters associated with learning for thousands of 
years, it is not yet two decades since the term ‘e-learning’ entered mainstream discourse. It is therefore 
important to make explicit what is meant by this term as it has been appropriated by diverse communities of 
practice since it first appeared around 1998-99 (Cross, 2004; CIPD, 2008; Garrison and Anderson, 2003). 

e-Learning can signify both a theoretical discourse and a range of activities that take place in many 
contexts - formal and informal, within educational institutions and workplace settings, or elsewhere ‘any 
time any place ’ as the saying goes. Adopters of the term include corporate training associations, professional 
associations, academic Web enthusiasts, government policy makers, software vendors, standards 
development organizations, and military organizations, just to name a few (Mason, 2005:320). There are 
distinctions according to context. For example. Bates identifies key differences between post-secondary 
education and corporate settings - the latter being more concerned with the broader context of knowledge 
management, the former focused on learning and research (Bates, 2004: 275). In an attempt to broaden 
philosophical perspective, Friesen puts the case for “re-thinking e-learning research” and argues for a 
“reconceptualization of e-learning as an inter- and cross-disciplinary endeavor” (Friesen, 2009:20). 
Conceptualizing in even broader terms. Cooper argues that its scope of activity is best understood as 
‘emergent’ and therefore subject to analyses that highlight perspectives on “complexity” (Cooper, 2010). 
Others prefer to use the related terminology ‘online learning’ to frame the challenges of “integrating 
technology into classroom instruction” (Tomei and Morris, 2011). For the purposes of this paper, e-learning 
is defined as: learning that is facilitated by engagement with ICT. 

2.2 Innovation and Practice 

With the above definition in mind, a diversity of ICT development and innovation over the last few decades 
can meaningfully be described as e-learning systems, environments, or platforms. Examples of structured , 
contained, or purpose-built platforms include computer-based training systems, learning management 
systems, intelligent tutoring systems, e-portfolio systems, performance support systems, virtual worlds, 
gaming environments, e-books, and other related applications and services. Anyone with a young child who 
has access to an iPad will also know how engaging and educational a single app can be - whether it is 
explicitly educational or not. Examples of unstructured and open environments that can function as e- 
learning environments include use of mainstream search engines and social media. Benefits and deficiencies 
can be identified with all of these developments, as is documented in the extensive and growing discourse on 
e-learning - for example, the number of peer-reviewed journals worldwide dedicated to the subject is now in 
excess of 50 titles and the majority of these titles have emerged in the last five years. If related topics such as 
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Distance Education, e-Research, Technology in Society, Knowledge Creation, and Performance Support are 
included then there are hundreds of relevant journals. 

As e-learning develops into an established academic field it brings with it a discourse that refines its core 
concepts and terms while ICT innovations and trends evolve. It is also likely that certain trends and biases 
will be revealed along the way. For instance, evidence suggests that much of the first generation of practice 
associated with e-learning has been very focused on the delivery and access to purpose-built learning 
content , not so much with learning activities or the cognitive processes associated with learning (Dalziel, 
2003; Alonso, et al., 2005; LETSI, 2008; ADL, 2009). This first generation of learning content has also been 
constrained by metadata that is descriptive in function - in other words, metadata that describes the content in 
terms of semantics that have roots in who, what, when, and where. 

It is also the case that the educational potential of existing, emerging and future developments in ICT is 
now commonly discussed in many diverse settings (daily newspapers, school curriculum support materials, 
political party policy documents, workplace human resource departments, standards-setting bodies, academic 
literature, and in higher education strategic planning). The ‘ Digital Education Revolution’ policy of the 
Australian Governments during 2007-2012 is a prominent example of a public policy response. Such public 
policy has been commonplace since the invention of the World Wide Web, although prior to this, the 
transformative potential of educational technology was recognised at various other historical moments (such 
as with the inventions of radio, television, personal computers, interactive and game-based digital media). 
There are therefore multiple perspectives that help explain the history and viable developmental paths of e- 
leaming into the future. The Australian School of the Air , which began in 1951 and continues today in 
servicing the needs of remote communities in Australia, represents an example of an older communications 
technology that is still used effectively for educational purposes. This is significant because it suggests that 
the viability of a technology is not necessarily made redundant by new technologies. 

2.3 Historical and Social Narratives 

Broader historical perspective provides further context. Not only has evolution of the World Wide Web taken 
place within a short period of time accompanied by rapid innovation, it has been transformative, representing 
a global revolution in the production, distribution, and access to information and communications (Castells, 
1996; Benkler, 2006; Gleick, 201 1) and can be seen as having enormous impact upon teaching and learning. 

A number of commentators have consequently introduced narratives on the evolution of the Web in terms 
of its impact upon learning. Taylor (2001), for instance, began visioning “fifth generation distance education” 
around 2001-2002 as an “intelligent flexible learning model” - it was student-centric in conception but 
impacted significantly the organizational structures and readiness for institutions concerned. In 2005, 
Siemens proposed a new learning theory called “connectivism”, motivated principally by the impact of the 
proliferation of networked ICT applications and the limitations of dominant learning theories (behaviourism, 
cognitivism, and constructivism) to explain and support the scope of interactions a learner. The 
distinguishing characteristic of Siemens’ theory is the prominent role of networks in creating connections 
between disparate learning sources and events (Siemens, 2005). Siemens' work resonates with the extensive 
sociological work of Castells (1996, 2001) in outlining the “rise of the network society” and in the work of 
Benkler on the social production of intellectual capital (Benkler, 2006). 

More recently, there has been popular usage of the terminology ‘Web 2.0’ typically to describe 
networking capabilities that leverage social media providing individuals with enormous scope for publishing 
content and social interaction. Adoption of such terminology has also led to characterisations of “Learning 
2.0” being learning that is facilitated by Web 2.0 social media applications (Brown and Adler, 2008) and 
related commentary about the “post-LMS era” (Mott, 2010). The utility of such characterisations is yet to be 
determined; however, in terms of the evolution of e-learning, they can be somewhat misleading because they 
mask, or do not always explicitly acknowledge, the capabilities that already existed in early phases of 
development - such as in Computer Based Training (CBT), Computer Assisted Learning (CAL), Computer 
Managed Learning (CML), Computer Mediated Communication (CMC), and Computer Supported 
Collaborative Learning (CSCL). The important observation here is that there are numerous technologies that 
have shaped what e-learning is today. Secondly, and most importantly for this investigation, none of the 
innovations mentioned hitherto have explicitly explored how wM-questioning during learning might be 
explored, supported or scaffolded. 
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2.4 Into the Future 

With the foundations of e-learning now well-established there is enormous scope for new developments that 
may enrich learning experiences through supporting deeper inquiry and cognitive engagement via 
environments that stimulate reflective practice and the development of understanding while learning online. 
A number of likely future trajectories can be discerned from the current context - for example, the broad 
uptake of social media provides stimulus for the use of diverse collaborative environments at scales 
unprecedented. Other developments will emerge as a consequence of ubiquitous broadband connectivity, 
innovations in natural language search technologies, access to open content, the proliferation of mobile 
technologies, work integrated learning programs, and intelligent tutoring systems. Will IT develop further as 
an “intelligent technology” or an “interruption technology” (Carr, 2010)? No doubt, unexpected innovations 
will also impact the evolutionary story. 

This paper, however, is concerned with one of the frontiers that beckon further development - ICT that 
supports deep learning instigated by questioning , reflective practice, and promotes cognitive engagement. 


3. COGNITIVE ENGAGEMENT 

3.1 Ubiquitous Distraction? 

There can be little doubt that the Internet has spawned a proliferation of ICT tools useful for learning. But the 
story of the impact of such relentless innovation is not an intrinsically positive one. It is also accompanied by 
a growing discourse arguing that extended use of the Internet can also have detrimental effects on cognition 
and behaviour (Clark, 2002; Bannister and Remenyi, 2009; Carr, 2010; Aguirre, 2011; Chalupa, 2011). 
Evidence shows there is definitely an impact upon cognitive load (Verhoeven, 2009; Kleinberg, 2011), a 
topic that instructional designers have been concerned with for decades (Sweller, 1994). For example, for 
reasons that being online can be very distracting with the effect of weakening cognitive focus, the term 
“interruption technology” has been a catch-phrase in contemporary popular commentary on the Internet: 

the single most mind-altering technology that has ever come into general use . . . when we go online, 
we enter an environment that promotes cursory reading, hurried and distracted thinking, and 
superficial learning ... The Net’s cacophony of stimuli short-circuits both conscious and 
unconscious thought, preventing our minds from thinking either deeply or creatively. (Carr, 2010) 

Of course, similar commentary and research has existed for decades about extended exposure to 
television and virtual gaming environments. Thus, the discourse is not all negative - for example, research 
shows that while extended Internet use can cause some loss of short-term memory there is also a gain in that 
“The Internet has become a primary form of external or transactive memory, where information is stored 
collectively outside ourselves” (Sparrow, et al., 2011). 

There is truth in both arguments - so in terms of the nature of cognitive engagement while learning 
online, evidence that drives this debate will be important for researchers to track. 

3.2 The Search Paradigm 

The enormous market success of the Google search engine can be seen as paradigm-shaping in the way that 
much learning online and scholarship is now initiated - via search. Its functionality has also delivered routine 
information retrieval and discovery into the mainstream. Of course, not all searches using Google are 
concerned with learning and most are better classified as information-seeking and Google’s effectiveness has 
also impacted corporate workflows, the socialization of information (Brown and Duguid, 2000), 
Government-based services, and the expectations of citizens of the developed world. As Google (the 
company) has developed its own services, such as Gmail and Google Docs and Drive, the flagship search 
engine can be seen as the core piece of technical architecture - search being the key operator on, and 
organizing technology for, content. Again, however, Carr notes a downside: 

Google . . . shapes our relationship with the content that it serves up so efficiently and in such profusion. 
The intellectual technologies it has pioneered promote the speedy, superficial skimming of information 
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and discourage any deep, prolonged engagement with a single argument, idea, or narrative. “Our goal,” 

says Irene Au, “is to get users in and out really quickly. All our design decisions are based on that 

strategy.” (Carr, 2010:156) 

The immediate counterpoint to this argument is that innovations in ICT are far richer than the Google 
suite of services. But, there is a further issue with the ‘Google paradigm’ relevant here: its search engine is 
calibrated with a design bias that privileges the aboutness of content - in other words, it is focused on parsing 
information as data. Its internal indexes are all built on data that is factual and measurable ; and searches are 
typically instigated by keywords and phrases, not questions constructed in natural language. Thus, 
interactions with Google can be seen as being constrained by “factoid" information (Verberne, 2010), or 
what Mason describes as the “primitives of information-retrieval” - facets of information that are readily 
associated with questions of who, what, when, and where (Mason, 2008). While Google uses sophisticated 
algorithms involving various weightings associated with “backlinks” this still functions as factoid 
information. Even with value-added services to Google search, such as ManagedQ, results to queries are 
organized into sets associated with people {who), things (what), and places (where). This underlying 
constraint has the effect of ‘information begetting information’ and interrupts prolonged inquiry or direct 
pathways into the discovery of content that is explanatory in nature (Mason. 2008; 2011a). This does not 
mean that explanatory content is not retrieved, just that it is not easily or directly discovered. In particular, 
queries that are conceived with ‘why’ in mind are not parsed well by Google because of the semantic 
ambiguity and linguistic versatility of the term why (Evered, 2005; Verberne, 2010; Mason, 2008). This has 
significant repercussions for the design of ICT systems aimed at supporting learning. 

3.3 Dimensions of Why -Related Research 

Why distinguishes itself from other ‘primitive’ questions (who, what, when, where, and how), in that it often 
requires a plausible explanation or rationale as an adequate response - in other words, reasoning as well as 
information (Verberne, 2010:10). Thus, w/zy-questioning has the potential to initiate a shift from information 
processing to engagement of other cognitive functions, such as inquiry, analysis, problem-solving, and 
reflection. As Walton has noted, why is a key initiator of dialogue (Walton, 2004). 

For researchers pursuing question-generation techniques in intelligent tutoring, why questions are seen to 
belong to a “deep/complex” category of all possible question types (Graesser, et al., 2007). Evered (2005) 
provides an analysis in which the function of responses to w/zy-questioning is categorized according to three 
classes of explanation: Causal (Why E? Because C (C = cause)); Teleological (Why E? In order to P (P = 
Purpose)); and Gestaltic (Why E? For these reasons, R (R = Reasons)) (Evered, 2005:201). Thus, in 
identifying opportunities for ICT-enabled scaffolding that might support inquiry and reflection, access to and 
production of explanatory content, as distinct from descriptive content, is of prime concern here. 

It is also interesting, however, that while why can be shown to have wide linguistic versatility (Mason, 
2011a:93) it is not regarded as a “semantic prime” by linguists developing Natural Semantic Metalanguage 
(research that is focused on identifying concepts with irreducible semantics), primarily because this 
versatility is not free from ambiguity (Goddard and Wierzbicka, 2007). 

Thus, in probing the linguistic dimensions of why, at least five key activities relevant to e-learning can be 
identified - asking, learning, knowing, understanding, and explaining why. The literature on educational 
psychology tells us that asking why is an important foundation of inquiry and fundamental to the 
development of reasoning skills and learning (Dewey, 1966; Piaget, 1966; Schank and Cleary, 1995; Bruce 
and Casey, 2012). Processes of learning, knowing, and understanding why build upon inquiry and all involve 
reflective practice (Schon, 1987:72). After learning something, explaining why can reveal a person's 
understanding (or lack of it). Thus the motivating question for this investigation: what ICT scaffolding - as 
application, services, or interventions - might support inquiry instigated by w/zy-questioning? 

3.4 Tools for Scaffolding and Reflective Practice 

Investigations into ICT tools that explicitly aim to support w/zy-questioning reveals a number of search 
technologies based upon natural language processing and computational linguistics, although findings to date 
demonstrate that much research is yet to be done (Ferrucci, et al., 2010; Verberne, 2010). Research is also 
proceeding in the fields of information science (metadata schemas and question-answer techniques) and 
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question-generation for intelligent tutoring (Kunze, 2001; Mason, 2008; Rus and Graesser, 2008). Of 
immediate relevance, however, is the application of wikis and e-portfolio systems to support reflective 
practice that is consistent with the goals of inquiry-based learning. Evidence is mounting that both 
approaches - one via the route of enlisting open, social engagement in content production (wikis); the other, 
individually-controlled reflective journalism that is discretionally shared - develop reflective practice and 
therefore prolonged cognitive engagement (Loo, 2012; Mason 2011b). A challenge, then, that is specific to 
the focus of this investigation is how scaffolding interventions might leverage these platforms. 


4. CONCLUSION 

Investigations into w/ry-questioning reveal there are significant repercussions for the design, development, 
and utilization of ICT systems aimed at supporting learning. In particular, accommodating multiple 
dimensions of why - asking, learning, knowing, understanding, and explaining - point to a frontier that will 
focus on the pivotal role of explanatory content and prolonged cognitive engagement through reflective 
practice. 
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ABSTRACT 

This paper deals with two types of logical problems - recognition problems and reverse engineering problems, and with 
the interrelations between these types of problems. The recognition problems are modeled in the form of a visual 
representation of various objects in a common pattern, with a composition of represented objects in the pattern. Solving 
the recognition problem may therefore be understood as recognizing a visually-represented Boolean concept, with further 
formulation of the concept. The recognition problems can be perceived as a parallel process, so the recognition problems 
are considered a parallel type. Alternatively, solving a reverse engineering problem means reconstructing a Boolean 
function/concept implemented within a given “black box”. Since such a reconstruction is typically performed 
sequentially, step by step, this type of problem can be considered a sequential type. We study the above two types of 
problems for the same set of Boolean concepts and compare the corresponding solutions obtained by a group of students. 
The paper presents results of experiments that study how the complexity of Boolean concepts affects the students’ 
success in solving parallel and the sequential type problems respectively. 


KEYWORDS 

Boolean concepts. Recognition and Reverse engineering problems. 


1. INTRODUCTION 

Concepts are the atoms of thought and they are therefore at the nucleus of cognition science (Fodor, 1994). 
People begin to acquire concepts from infancy and continue to acquire and plan new concepts throughout 
their entire lives (Medin, Lynch, & Solomon. 2000; Medin & Smith, 1984). One way to create a new concept 
is by utilizing existing concepts in different combinations. One of the problems in learning concepts is 
determining the concept’s subjective difficulty. Logical thinking is the key to a wide variety of complex 
problem solving and decision making processes and therefore Boolean concepts are essential. An important 
aspect of concept learning theory is the ability to predict the level of difficulty in learning different types of 
concepts. In this respect. Boolean concepts are a fundamental topic in the literature. Boolean concepts can be 
defined by a Boolean expression composed of basic logic operations: negation, disjunction (“or”), and 
conjunction (“and”). These types of Boolean concepts have been studied extensively by Shepard, Hovland, 
and Jenkins-SHJ (1961), (Bourne 1966, 1974), (Nosofsky, Gluck Palmeri, McKinley, and Glauthier 1994). 
These studies focused on Boolean concepts with three binary variables, where the concept receives “1” for 4 
out of 8 possible combinations and "0” for the remaining 4 combinations. Since some of the 70 possible 
Boolean concepts are congruent, they can be categorized as the same type into six subcategories. The six 
subcategories with structural equivalence can be described in a geometrical representation using cubes 
(Figure 1). 

This notion of congruence seems to have been first introduced by Aiken and his colleagues (Aiken & the 
Staff of the Computation Laboratory at Harvard University, 1951) and subsequently became prevalent in the 
literature on the theory of switching circuits. It was introduced into psychology by Shepard, Hovland and 
Jenkins (1961). 
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Figure 1. Shepard, Hovland, and Jenkins (SHJ) category types. 

1.1 Cognitive Complexity of Boolean Concepts 

Concept subcategories with structural equivalence belong to the same category and are defined as a Type. 
The study results pertaining to the six types of concepts presented in Figl showed that the concepts 
belonging to Typel are the simplest to learn and the subgroup of concepts belonging to Type6 are the most 
difficult, according to the following order: Type l<Type2<(Type3,Type4,Type5)<Type6. The results of this 
study are highly influential since SHJ proposed two informal hypotheses, the first being that the number of 
literals in the minimal expression predicts the concept’s level of difficulty. The second hypothesis is that 
ranking the difficulty among the concepts in each type depends on the number of binary variables in the 
concept. The concept subcategory in Typel has one variable, the subcategory concept in Type2 has only 2 
variables, and concept subcategories in Types2 to 6 contain three variables. 

Feldman (2000), based on the conclusions from the SHJ study, defined a quantitative relationship 
between the level of difficulty of learning Boolean concepts and the concept’s Boolean complexity. 
Assuming that D is the number of binary features and P is the number of combinations out of all the 
combinations in which the Boolean concept receives “1” (SOP-Sum Of Products), D[P] indicates the family 
of Boolean concepts with D variables and P combinations where the concept receives “1”. For example: the 
concepts that were examined by SHJ were represented as 3 [4], 3 binary variables and 4 combinations in 
which the concept receives “1”. In his study, Feldman examined the 3 [2], 3[3], 3[4], 4[2], 4[3] concept 
family. Feldman also addressed the family of concepts where the number of combinations with “1” differs 
from the number of combinations with “0”. For example, the concept 3 [3] contains 3 combinations with “1” 
and 5 combinations with “0”, unlike its mirror concept 3[5], which contains 3 combinations with “0” and 5 
combinations with “1”. The complexity measure of a Boolean concept as defined by Feldman is the number 
of literals in the most minimal expression that represents the concept’s complete SOP. According to 
Feldman’s definition of the complexity of the concepts and use of heuristic minimizing technique to the 
minimum literals, in SHJ’s classification model the complexity measures in each category are: Typel: 1, 
Type2: 4, Type3: 6, Type4: 6, Type5: 6, Type6: 10. According to this complexity measure, it is possible to 
predict that concepts from Types3, 4, 5 have the same complexity and are learned more easily than Type6 but 
are more difficult to learn than Types 1 and 2. These complexity measures predict difficulties in learning 
Boolean concepts, as examined by SHJ. 

Since there are several techniques for reducing an expression to the minimum, Vigo (2006) presented use 
of the QM ( Quine-McCluskey) technique to obtain a correct minimal description and showed that it is 
possible to minimize the expressions more correctly than what Feldman presented in the heuristic technique. 

According to Vigo, based on Feldman’s same definition as a measure of cognitive complexity in SHJ’s 
classification model, the complexities of functions from each type are given in Table 1. 

The definition of the Boolean concept’s complexity as a minimal number of literals in a minimal 
expression creates several problems. The first: because the complexity is defined as the number of literals in 
the minimal expression and the expressions can be minimized using several techniques, a uniform 
complexity measure cannot be obtained. For example: according to Feldman’s heuristics, Types3, 4, 5 have 
the same complexity. Contrary to Feldman, in a more correct minimal expression according to Vigo, 
concepts from Types2 and 3 have the same complexity. 

The second problem: studies show that the Boolean concept “xor” as an operator is learned and acquired 
as a concept in human thought to the same degree or more easily than the Boolean concept “or” as an 
operator (Evans, Newstead, & Byrne, 1993). By using the “xor” operator, some of the Boolean expressions 
examined by Feldman and Vigo can be simplified significantly and therefore, the complexity decreases 
according to the definition as a minimum of literals in the minimal expression. In light of the problems 
presented, Feldman and Vigo developed alternative theories for Boolean complexity as a measure of 
predicting the difficulty in learning Boolean concepts. 
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Table 1. The six concepts in SHJ and their descriptions according to Feldmans heuristic, correct minimal descriptions, 
minimal descriptions using "xor” and Structural Complexity (SC) 


The SHJ six 
concepts 

Feldman's 

heuristic 

correct minimal descriptions 

minimal descriptions 
using “xor” 

SC 

Typel 3[4 ] 

a( 1) 

a (1) 

a (1) 

1.66 

Type2 3[4 ] 

ab + a b (4) 

ab + ab (4) 

a ® b (2) 

2.00 

Type3 3[4] 

a (b c) + a b c (6) 

a c + b c (4) 

a c + b c (4) 

2.14 

Type4 3[4 ] 

a (b c) + a c (6) 

c(a + + fl b (5) 

c[ci + b^j + a b (5) 

2.14 

Type5 3 [ 4 ] 

a (b c) + a b c (6) 

a (b c ) + a b c (6) 

a® (be) (3) 

2.34 

Type6 3[4] 

a [b c+c b ) + a {b c + b (10) 

a ( j ) c+c bj + a (b c + b c) (10) 

a®b®c (3) 

4.00 


Aware of these difficulties, Feldman (2006) has recently introduced his spectral decomposition model. In 
this updated model, the complexity of a concept is driven by its decomposition into a set of underlying 
regularities. The basic idea is that learning from examples involves the extractions of patterns and 
regularities. The formal model describes how a pattern (expressed in terms of a Boolean rule) may be 
decomposed algebraically into a “spectrum" of component patterns, each of which has a simpler or more 
“atomic" regularity. Regularities of a higher degree represent more idiosyncratic patterns while regularities of 
a lower degree represent simpler patterns in the original decomposed pattern. There are two kinds of simple 
concepts: those that consist of a single constant value of a variable, and those that consist of an implication 
between the values of two variables. These two basic types of concepts can be algebraically combined to 
represent more complex linear concepts. This step is based on an analysis of a concept's “power spectrum”. 
Thus, any concept can be decomposed into a set of underlying rules, each of differing degrees of complexity, 
depending on the number of variables that they instantiate. The weighted mean of the complexity of these 
underlying rules then provides an overall index of the algebraic complexity of the concept. The model yields 
a complexity index for any Boolean concept. This model is characterized by its ability to explain unique 
phenomena of the learning process of concepts. The algebraic complexity makes it possible to rank each 
concept combination according to its structure. 

Vigo (2009), developed an alternative theory for calculating the complexity measure of a Boolean 
concept, defined as structural complexity. The theory is based on a Boolean derivative. The Boolean 
derivative was introduced by Reed (1954) in a discussion of error-correcting codes in electrical circuits. The 
basic concept has been mainly relegated to this very specialized domain of applied Boolean algebra. 

The question that the theory is supposed to address is: What is it about the internal structure of Boolean 
concepts from any category that makes them harder to learn than concepts from a different category? For the 
purpose of quantitative definition of the structural complexity, the degree of categorical invariance must be 
calculated. In order to quantify the structural complexity, several calculations that will be presented below 
must be conducted. For a Boolean function with n variables derived from the Boolean part similar to the 
partial derivative of a mathematical function defined as follows: 


dF(x l ,x 2 ,..x i ,..x n ) 


dx , 


= F(x 1 ,x 2 ,..x i ,..x n )® F[x v x 2 , 


Let F be a Boolean expression that defines a Boolean category. The logical manifold or Lm of F in 
respect to x is defined as: 

(M F rl) 


67 




ISBN: 978-989-8533-12-8 © 2012 IADIS 


were: 


F 0 - min terms sum of (F (*)) , F pi - max terms sum of 


f 8F(x)) 

( dF (.*)" 

{ dx. J 

1 dx > J 


The degree of structural complexity (SC) of a Boolean category defined by the Boolean expression F and 
belonging to the family DfP] is indirectly proportional to its degree of categorical invariance and directly 
proportional to its cardinality SOP(F): 


» .12 

( L m i ) + 1 


Vigo’s (2009) account of the invariance of concepts, as he acknowledges, does not specify how 
individuals learn concepts. He suggests only that cognitive processes could detect invariances by comparing 
a set of instances to the set yielded by the partial derivative of each variable. For SHJ's six categories, the 
complexity measures described above appear in Table 1. 

1.2 Recognition Problems 

The recognition problems are modeled in the form of visual representation of various objects in a common 
pattern, with composition of thus represented objects in the pattern. Solving the recognition problem may 
thus be understood as recognizing a visually-represented Boolean concept, with further formulation of the 
concept. The recognition problems can be perceived as a parallel process, so the recognition problems are 
considered of a parallel type. 

One of the important roles of human consciousness is to reveal patterns and find data sequences. Not all 
the patterns leave the same impression on people as a basis for identification and perhaps subsequently, 
identical patterns are not equally observed by different people. 

1.3 Reverse Engineering (RE) 

The process of finding and reconstructing operating mechanisms in a given functional system of a digital 
electronic apparatus is defined as Reverse Engineering (RE) (Chikofsky & Cross, 1990). RE is applied in a 
wide variety of fields: competition in manufacturing new products, from electronic components to cars, 
among competing companies without infringing upon the competing company’s copyrights, replacing system 
components with refurbished or spare parts (Ingle, 1994), solving problems and defects in a system (Eilam, 
2005). RE can be referred to as a certain type of problem solving. 

A reverse engineering problem means reconstructing a Boolean function implemented within a given “black 
box”. Since such a reconstruction is typically performed sequentially, step-by-step, this type of problem can 
be considered a sequential type. 

When the values in the variables in a Boolean function express declarations or claims in specific 
combinations, the result of the expression is not a Boolean concept but “true” or “false”. For example: 
“Switch one and switch two are on or switch three is off’. People are capable of making such claims by 
examining the combinations. Sequentially examining a series of combinations makes it possible to reveal all 
the possible combinations that give a “true” or “false” value. A mental model for the possibilities that give a 
“true” value (a mental model demonstrates the “true” principle) has been built based on a discovery of the 
combinations. The mental model is translated into a representation of the system using literals, by 
minimizing the variables that do not affect the “true” result (Johnson-Laird, 2006). 

This paper deals with two main questions. The first: What effect does the cognitive complexity of a 
Boolean concept have on the success of solving recognition and reverse engineering type problems? What is 
the ratio between the cognitive complexity of a Boolean concept and the complexity of solving recognition 
and reverse engineering type problems? 

Reverse engineering problems were represented as a black box that can be used to control the lighting of 
a bulb using independent switches. Recognition problems were given as a pattern containing geometric 
shapes. 
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We study the above two types of problems for the same set of 9 Boolean concepts and compare the 
corresponding solutions obtained by a group of students. 


2. EXPERIMENT 

The experiment was conducted in two stages for 9 concepts, where each concept was described by means of 
a Boolean expression in Table 2. 

During the first stage, RE problems were examined using a black box that could be used to control the 
lighting of a bulb using three independent switches for seven concepts with three variables (concepts 1-7, 
Table2) and using four independent switches for two concepts with four variables (concepts 8 and 9, Table2). 
The participants were required to try the different switch combinations that light the bulb and describe the 
combinations that light the bulb for each of the 9 concepts using a Boolean expression. A maximum of 5 
minutes was allocated for each of the 9 tasks. The tasks were presented as a simulation on a computer 
monitor and the time taken to complete each task was measured. Successful completion of the task was 
measured based on correct solving during the allotted time. 

During the second stage, recognition problems were examined using a questionnaire with nine patterns, 
where each pattern represents one of the nine concepts examined, respectively, Figure 2 presents patterns for 
concepts 1, 5 and 8. The participants were asked to describe each of the patterns with as few literals as 
possible. A maximum of 45 minutes was allocated to completing the questionnaire for each of the nine 
patterns. The two stages of the experiment were conducted two days apart. 






A O A © 

A 0 A 




° O 

0 # ® A 

A 


PN-I PN-S PN-S 


Figure 2. Patterns for concepts 1, 5 and 8 respectively tested during the experiment. 

Successful completion of the tasks solving both RE problems and the recognition questionnaire was 
measured based on correct solving in the allotted time. All the solutions were examined compared to two 
complexity measures: complexity according to a minimal representation of the expression (including using 
the “xor” operator) and complexity according to structural complexity (see Table2) 

An example for calculating structural complexity for concept 1 (PN-1 in Table2): 


F 10 (a,fc,c) = £> (a + c) = ^ (l,4,5) 
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2.1 Method 

The research population included thirty 1 st year students studying for a Bachelor of Engineering degree at a 
college. Twenty students studied in the Department of Electric and Electronic Engineering and 10 students 
studied in the Software Engineering Department. All students studied the Logic Design course in the same 
study group and with the same lecturer. All students completed the course successfully with the final exam 
average grade of 75. 


Table 2. The 9 concepts were tested during the experiment and their descriptions according to correct minimal 
descriptions-MD, minimal descriptions using “xor”-MD (xor) and Structural Complexity (SC) 


PN 

FCN 

MD 

MD (xor) 

sc 

Success (%) 
RE problems 

Success (%) 
Rec. problems 

1 

Typel 3 [ 3 ] 

b (a + c ) (3) 

- 

1.54 

100 

100 

2 

Type 3 3 [ 4 ] 

a c + b c (4) 

- 

2.14 

90 

70 

3 

Type 3 3 [ 4 ] 

a b + ab (4) 

a®b (2) 

2.14 

100 

80 

4 

Type4 3[4] 

a (fe + c) + be (5) 

- 

2.14 

100 

90 

5 

Type5 3[4] 

(a + b^c + a b c (6) 

c © ( ab ) (3) 

2.34 

60 

40 

6 

Type 6 3[4] 

a[b c+c b^j + a(b c + b cj ( 10 ) 

a®b®c ( 3 ) 

4.00 

40 

30 

7 

Type 6 3 [ 4 ] 

a[b c+c b ^ + a{b c + b cj ( 10 ) 

a ®b®c ( 3 ) 

4.00 

60 

30 

8 

Type4 4[5] 

a(b + c + d) + b(d +c) + ccl (9) 

' 

4.48 

90 

80 

9 

Type4 4[5] 

a(b + c + d^ + b{d -t-cj + c d (9) 

- 

4.48 

90 

70 


An experimental environment using Lab View was developed on a computer monitor for “black box” RE 
problems. The monitor displayed a black box with switches and a bulb. The state of the switches could be 
changed by clicking on the appropriate key with the mouse. A change in the switch’s state resulted in a color 
change from black to red and the written indication “off’ (black) and “on” (red). According to the switch's 
state, the light bulb is either on or off. A lit bulb is green and has the word “on”, and a bulb that is off is with 
the word “off’. As soon as the participants reached the conclusion that they knew the appropriate logical 
function for the system of switches and the bulb, they were asked to write the states of the switches that light 
the bulb using a Boolean expression, written using an equation generator, and to press “finish”. Time was 
measured from the moment the first switch in the box is pressed until the task was completed. The course of 
the experiment was filmed and a brief interview was conducted at the end of the task. The objective of the 
interview was to examine the solution strategy employed by the participants to discover the functionality 
regarding each of the nine tasks. The video clips were analyzed to compare the strategy that the students 
stated during the interview and the solution strategy as observed in the video clips. 

A two-part questionnaire was developed for recognition problems. The first part presented a pattern with 
eight geometric shapes. Large and small triangles in red and blue, and large and small circles in red and blue. 
The shape was defined using binary variable a , size was defined as binary variable b and color was defined 
using binary variable c. Seven patterns were then presented, where each of the patterns matched the 
examined Boolean concept (Table2), respectively. The second part presented a pattern with 16 geometric 
shapes. Large and small triangles in red and blue and large and small circles in red and blue, with a nucleus 
in the center of the shape and without a nucleus. The shape was defined using binary variable a, the size was 
defined using binary variable b, the color was defined using binary variable c, and the nucleus was defined 
using binary variable d. Time was measured from the commencement of solving the questionnaire to when 
each participant submitted his questionnaire. 
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3. RESULTS AND CONCLUSION 

By examining the results and conclusions, we will assess the effect the various complexity measures as 
presented in Table 2 have on the success, or lack thereof, in solving the two types of problems - recognition 
and RE problems, and the relationship between them. Additional aspects that we will examine are the effect 
of the “xor” operator on complexity, the “true principle” in RE problems, meaning whether the tendency is to 
focus on combinations that give “1” also for problems in which the number of combinations whose result is 
“0” is significantly less than the combinations that give “1”. 

Table 2 presents the success rates for solving RE problems and recognition problems for all the nine 
concepts presented in the Table. The results show that participants are more successful in solving RE 
problems than solving recognition problems. Not a single participant that did not succeed in solving RE 
problems managed to solve recognition problems for the same concept. However, not all the participants that 
managed to solve RE problems were also successful in solving the recognition problems. For example, for 
problem 3, 27 participants managed to solve the RE problem, compared to 24 of them that also successfully 
solved the recognition problem. Difficulty in comprehending and learning a Boolean concept cannot be 
predicted based on Boolean complexity measures of the concept alone, but rather it also depends on the type 
of problem representing the concept. It can be concluded from the results that for problems where the 
information is absorbed concurrently, recognition problems are more difficult to learn than problems where 
the information is sequentially obtained, in this case RE problems. Therefore, the difficulty not only depends 
on the concept’s complexity but also on the complexity of the manner in which the problem is presented. 

The greater the complexity measure, the lower the success rates, except for PN4, PN8 and PN9, which we 
will elaborate on later. It cannot be unequivocally concluded that the greater the complexity measure the 
lower the success rate for solving the problem. For problems 1, 2-3, 5, 6-7, the success rates for solving both 
types of problems indeed decreases as the complexity measure increases, also according to the MD and SC 
measures. For problem 1, the complexity is the lowest and the success rate the highest. For problems 2 and 3, 
the complexity is slightly higher than problem 1 and the success rates are consequently smaller. The situation 
is the same for problems 6, 5, and 7, respectively. The inverse relationship between the complexity and 
success rate also manifests in the average time required to solve the problems. As the complexity increases, 
the average time it takes to solve the problem increases accordingly. 

Complexity with minimal literals when the “xor” operator is involved is a measure that is the least 
predictive, relative to MD and SC, of the subjective difficulty of successfully solving the problem. The 
majority of participants did not recognize the “xor” operator in both types of problems. Participants that 
grasped the “xor” concept as an operator to the same degree as the “or” and “and" concepts were more 
successful in solving the problem and their average time was substantially lower. Eight out the thirty 
participants consistently used “xor” in problems 6, 3, and 7. All of the eight participants arrived at the correct 
solution for both types of problems at an average time of 30 seconds, which is much lower than the average 
time it took the total number of participants. Out of twenty two participants that did not use “xor” but that did 
use the other operators, sixteen participants succeeded in solving problem 3 as a recognition problem. For 
problem 6, four out of twenty two participants managed to solve the RE problem and nine out of twenty two 
managed to solve the recognition problem. For problem 7, ten out of twenty two managed to solve the RE 
problem and one managed to solve the recognition problem. It can be concluded from the results that if the 
“xor” operator is acquired as a concept to the same degree as the operators “or” and “and”, the concepts’ 
level of complexity decreases, the success rates increase, and the difficulty in solving the problem decreases. 
However, not everyone acquires the three concepts - “xor”, “or” and “and" to the same degree. For most, the 
“xor” concept is more difficult to grasp than the other two concepts. 

Although the level of complexity of the concepts in problems 4, 8, 9 are higher relative to problems 1, 2, 
3, the success rates for solving both types of problems is significantly higher and the average time needed to 
solve them is much lower. It can be hypothesized that the reason for this is that the concepts in problems 4, 8, 
9 fulfill qualities of symmetric functions. Apparently, the MD and SC complexity measures are not 
sufficiently reliable in predicting the level of difficulty in solving the problems for symmetrical functions. 
Based on the hypothesis, it is interesting to examine the effect of qualities of Boolean functions such as 
symmetry, linearity, etc. on the complexity of Boolean concepts. 

Two RE problem solving strategies were observed. The first, adopted by twenty seven out of the thirty 
participants, was to attribute a logical value to one of the variables and conduct fewer checks of the 
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combinations to a check times 2 consistently, building a mental model for the combinations in which the 
transitions among them, a change in one of the variables does not influence the state of the lit bulb. For 
problems 8 and 9, only ten out of twenty two participants reached the solution for the states in which the bulb 
is not lit and managed to solve the problem, since the number of combinations in which the bulb is not lit is 
4, versus 12 states in which the bulb is lit. They took the same approach with recognition problems in solving 
problems 8 and 9, and succeeded. For the remaining participants, the “true principle” guided them in solving 
all the states, including the ones where it is more effective to examine the states in which the bulb is not lit, 
which were significantly lower than the number for states in which the bulb is lit. Three students tended to 
use the strategy of covering all the possible states. They managed to reach the correct solution only for RE 
problems for concepts 1, 3, 4, and did not succeed in reaching the required solution for the other states. 
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ABSTRACT 

The foreign faculty in Kazakhstan faced the challenge of designing a total authentic learning curricula for undergraduate 
students used to a traditional approach to learning. Students were asked to submit reflective learning journeys at the end 
of the spring 2011 semester. This study uses their stories as a springboard to explore their perspective of authentic 
learning. The case study was confined to data gathered from one class, which revealed that those students who remained 
in the course wanted to be challenged by authentic learning. To them, authentic learning was both a force driving their 
evolution from passive to active learners and a means for solving the real-world problems with a community of co- 
learners. Authentic learning is an agent of change that is worth contemplating. Its design can apply to the multiple life- 
spaces of a single student and other disciplines. 


KEYWORDS 
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1. INTRODUCTION 

Kazakhstan (KZ) is the 9 th largest country in the world, with about sixteen million people and ninety higher 
education institutions. Newsweek (2010) ranked KZ 14 th worldwide in education, with a rating of 91.35% 
and a literacy rate of 99.5%. In Almaty, the KIMEP University has been ranked number one. 

Kazakhstani students are generally ambitious. For example, many MBA students would overload 
themselves concurrently in study and in work and would resort to complaining as a survival mechanism to 
cope with the unrealistic pressure (Misch & Galantine, 2009). 

A preference for rote learning (Lillis, 2009) appeared during the inquiry. It was further validated by first- 
hand experiences with students during the fall semester of 2010. Nevertheless, many students would want to 
be challenged by a demanding learning experience (Mclnnis, 2001). Therefore, authentic learning was 
introduced. However, it posed a huge challenge to faculty. 

1.1 How did the Study Come About? 

Mclnnis and Newsweek provided reasons to the belief that an authentic learning curriculum could be 
workable in an institution that claims to be world-class. As a consequence, the researcher incorporated 
authentic learning in all his classes in the spring semester of 2011. In this semester, he facilitated a Financial 
Statement Analysis (FSA) class (and two classes of Financial Accounting 2). The FSA was double-coded as 
ACC4208/FIN4214 for both senior and graduating Accounting- and Finance-students. Two sections ran 
concurrently from 17:30 to 19:30 every Tuesday and Thursday. 

In the first section, a Kazakhstani instructor utilized a traditional teaching approach. His course consisted 
of five short exams, two projects, and a forty-percent final exam covering the first twelve chapters of Penman 
(2010). In the second section, the researcher approached the course non-traditionally. 

The second section covered Chapters 13 and 14 of Stickney et al. (2007). All chapters of Penman and 
other chapters of Stickey are resources to thoroughly understand these two chapters in focus. The syllabus 
required each student to engage inquiry and to prepare evidence-based portfolio, which made up forty percent 
of the summative assessment. The learning portfolio included a project of student's own choosing, a 
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reflective learning journal, and a mind map. The project required an executive summary with spreadsheets 
and other pieces of supporting evidence. 

Students are empowered to work on the project alone or in small team. Moreover, they would transform 
themselves to be reflective storytellers and to see their higher education as a construction of personal and 
social stories. 

This second section went through several transition states. During the first two weeks, sessions blended 
traditional lectures (to introduce theories) with cooperative learning methods (to inspire participations). 
While exploring the strengths and interests of the FSA students, the researcher learned of a student who 
worked at one of the Big Four accounting firms. The student was allotted an entire session to share his 
knowledge about the valuation of firms in the real world. How students reacted, interacted, and responded to 
the presentation and to the idea of teaching themselves weighed against tradition. 

The time seemed ripe to advance the learning to the next level. Bonus projects were add. Similar to the 
40% learning portfolio, students were empowered to select their own topic(s) and to organize themselves as a 
community of learners. 

Each student was distributed a yellow card. Three questions on the front side formalized their 
commitments (1) to conduct independent active in-depth learning engagements; (2) to submit weekly 
reflective journals; and (3) to undertake evidence-based and inquiry-based project in lieu of a written final 
exam. If students answered No to any of the three questions, they were advised to transfer to the first section. 
The yellow card would also be stamped for attendance (the front side) and student participation (the back 
side), in order to encourage them to show up for class and to take part in the engagements. 

The important feature of the authentic learning curriculum was the no examination policy. The other sixty 
percent consisted of (1) participation in the two focus-group discussions; (2) weekly reflective journals; (3) 
eight assignments (some of which could be done in groups); and (4) an essay. Students were also asked to 
read the facilitator’s teaching philosophy and to write their own (5) learning philosophy. Rubrics were 
designed to rate the assessments. The two focus-group discussions took place two weeks into the semester 
and two weeks prior to its end. Students reflected on their learning points. They also had the opportunity to 
participate in (6) an in-depth interview with an officer in the Research and PhD department. All these, except 
for the bonus projects, were designed before the start of the spring semester. 

Assessing team work was a challenge. To avoid free loaders in the group, the team needed to detail the 
distribution of the jobs. Grade would be a one step lower if the distribution was equally divided among the 
members. In real life, it is unlikely that a work could be shared evenly. 


2. AUTHENTIC LEARNING: ITS ESSENTIAL ATTRIBUTES 

“The main point of authentic learning is to let students encounter and master situations that 
resemble real life” (Cronin, 1993, p. 79). 

Authentic learning originated two decades ago as a simple idea of apprenticeship (Resnick, 1987). Such 
situated learning (Collins, 1988; Collins et al., 1989) connects theoretical learning to practical learning. 

The learning engagemens in classroom setting presuppose more authenticity (Cronin, 1993, p. 79) or 
more situation likely to be encountered in real life. The essence of authentic learning in curriculum design 
explores and exploits all available opportunities. The mere use of authentic learning tools in a traditional 
learning curriculum does not make it an authentic learning curriculum. 

One essential feature of authentic learning is segmentation. The traditional learning is deeply engrained in 
many students. Authentic learning is radical. It disrupts the comfort of many students. The research of Brown 
and Venkatesh (2003) in innovation shows that sixteen percent are pro innovation and eighty four percent are 
against it. However, forty percent of those people who resist could be persuaded to reduce their negative bias 
and to accept the innovation in the end. Based of this insight, students who accept authentic learning belong 
to a special segment. 

Second is the conception as agent of change. Authentic learning connects the “in-here” knowledge 
(thoughts, beliefs, feelings, and dreams) with the “out-there” world (the physical and material domains). The 
connectedness presupposes a heightened “in-here” awareness of one’s own learning engagements when 
making sense of the “out-there” reality. It is useful to look at authentic learning from a deterministic or 
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instrumental perspective. In the first perspective, the learning is a force that imposes change and becomes 
part of a web of influences that is always in motion and constantly influences the community of learners. In 
the second perspective, the learning comes with perceived utilities or benefits that are a means to an end. 
which is the change. Any impediments could be mitigated by heightening the construction of meaning, the 
cognitive presence or the social presence during the four cognitive stages (Garrison et al., 2010; Garrison & 
Arbaugh, 2007). Wolf (2001) refers the change as a new alchemy of mind. 

The third feature concerns the knowledge construction stages (or the knowledge development stages). 
Constructing and experiencing knowledge passes four cognitive stage (Garrison et al., 2001). First of the 
learning experience is the Agenda-setting stage. It is marked by the perception of the relative newness and 
values (Rogers, [1962] 2003) of the new curriculum. Second, the Exploration stage facilitates the 
examination of problems and encourages a deep approach to learning. The ideas produced in this stage are 
constructed into personal meaning during the Integration stage. Finally in the fourth stage, the Resolution 
stage links the ideas and meanings and evidence and reasoning in a chain. The need for heightened cognitive 
and for heightened social presence comes the need for heighten thinking (Lesh et al., 1999; Rule, 2006), such 
as asking questions, conducting studies, drawing conclusions, thinking about thinking (metacognition), 
revising theories, and communicating results and discourse (Akyol and Garrison, 2011; Callison and Lamb, 
2004; Flavell, 1979; Renzulli et al., 2004; Vygotsky, 1978). 

Fourth, a conception of authentic learning is as constructed knowledge. According to constructionism 
(Papert, [1980] 1993), knowledge is constructed through interactions of the community of learners, the 
creation of social artefacts (e.g., projects), and the building of new knowledge on old (prior) knowledge 
through above four cognitive stages. Papert ([1980] 1993) uses the metaphor of communal property (Barron, 
2004; Brandsford et al., 2006). 

Fifth is empowerment, allowing students to make the choice of what, when, where, how, and with whom 
the learning occurs (Barron, 2004; Kirby & Goodpaster, 2006). This puts the student’s chosen focus in an 
individual personal frame (Kellouth, 2003; Rule, 2006) so that the learning activities remain student centered 
(Callison & Lamb, 2004; Maina, 2004, Renzulli et al., 2004). Empowerment also presupposes the diversity 
of learning pathways (Brandsford et al., 2006), with no single method for carrying out tasks (Bergeron & 
Rudenga, 1996). However, empowerment accepts the non-empowered dimensions, such as codified 
knowledge (concepts, facts, and procedures), learning strategies, and the learning environment. The latter set 
influences observation, coaching, and practice (Lave, 1988). 


3. RESEARCH METHODOLOGY 

The domains of this case study could be imagined as the three layers in an inverted triangle. At the widest 
level on top is the local setting of the higher education institution in Almaty, Kazakhstan. In the middle of the 
triangle is the complex phenomenon: the authentic learning of the FSA in the spring 2011 semester. The 
lowest level of the triangle is the abstract case. The fourteen learning journeys have highlighted four complex 
issues of authentic learning (1) as an agent of change, (2) as a process to construct knowledge, (3) as 
constructed knowledge, and (4) as empowerment. These four issues constitute the abstract case. 

To explore the case and to thoroughly understand its issues, the research uses a tiered approach (Clark & 
Linn. 2003). The approach avoids a pitfall of designing one huge multiple-tiered study to address every 
single intertwining issue. Nevertheless, it assures that the complex phenomenon meets the rigor, yields good 
data, and maximizes learning (Carver, 2006; Stake, 2005). 

Four important theoretical assumptions are essential to analyzing and interpreting the findings. First, 
authentic learning presupposes the cognitive readiness among individual students to form a community of 
learners, in small groups or in a whole group (the whole class). Second, because not all students want to 
undertake the authentic learning, the radical learning approach underscores the segmentation and the 
percentage of acceptance. Third, authentic learning cannot happen apart from the contexts. A curriculum 
developed for a certain student segment is not automatically applicable other students, other courses, and 
other higher institutions unless it is modified or reflected to fit the new contexts. A second implication of this 
assumption regards facilitators not as the source of expert knowledge but rather the source of 
interdisciplinary knowledge. Fourth, authentic learning assumes that students learn better if the learning is 
relevant to their needs. 
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Fourteen students included their learning journeys as a part of their learning portfolio. The journeys 
delved into their experiences, beliefs, emotions and thoughts and provide evidence that was otherwise not 
measurable nor explicit. The case study utilized these journeys as the primary data source. However, five 
journeys were highlighted in this paper. The other ( 1 ) nine journeys were used to triangulate the analysis. 
Other pieces of evidence used in triangulation included (Yin, 2009): (2) seven fifteen-minute in-depth open- 
ended interviews conducted by a colleague at the end of the semester (9 th to 1 1 th of May 201 1); (3) submitted 
assignments; (4) the students’ learning philosophy; (5) the two focus-group discussions; (6) the record books 
with their GPAs and English proficiency scores; and observations. 

The units of analysis, as indicated in the problem, corollary question, and the four issues in the structured 
top-down diagram (see Figure 1), were as follows: ( 1 ) the cognitive presence in the Community of Inquiry 
(Anderson et al., 2001; Garrison et ah, 2001; Rouke et ah, 2001); (2) the perceived attributes (Rogers, [1962] 
2003) triggering interest and commitment to authentic learning; (3) the stages in constructing knowledge; and 
(4) other necessary structures and procedures. The chosen units of analysis align with the three domains and 
call for slightly different modes of analysis and inferential perspectives in the Discussions section. A possible 
weakness is the inclusion of both structures and processes, but it is warranted by the exploratory nature of the 
study. The strength is the researcher’s personal curiosity about the case. This intrinsic interest has made the 
inquiry memorable, challenging, and fun. 

The foundation of the modes of analysis was the continuous contemplation of the evidence, which 
subsequently led to the emergence of insights (the theory). This general inductive approach balanced rigor, 
flexibility and iteration in all stages, from the reading of the raw data to the construction of the chain of 
reasoning. The analysis of qualitative data became simpler (Thomas, 2006). Issues were meshed to their 
common ground (Guba & Lincoln, 2005). During the thinking process, raw textual structural and procedural 
descriptions in the learning journeys were highlighted and/or summarized to underscore the experiences of 
the teaching and learning processes. The interpretations and findings were linked to the research objective. 

The authentic learning of the FSA course in the spring semester of 2010 was an imperfect, real, and 
complex phenomenon. The interpretation of the learning journeys was the key to unlock its truth (Denzin & 
Lincoln, 2005). Although these journeys convey many shades of truths (absolute truths, empirically adequate 
truths, approximate truths, and even partial truths), nevertheless the stories passed through validation, 
triangulation, and argumentation in Figure 1 . 


4. RESULTS - THE CONCEPTS OF AUTHENTIC LEARNING 
4.1 Authentic Learning as Segmentation 

The curriculum design in the second section was radical in many ways. First, there was the no examination 
policy. Second, each student had a yellow card. It was used to inspire participations and at the same time 
monitor attendance. Third, the facilitator and the students played the role of co-teachers and co-learners in a 
community of learners. Thus, the teaching and learning passed through transitional stages. 

Whereas traditional learning is deeply engrained in many students, authentic learning disrupts the comfort 
of many students. Thus from initial enrolment of sixty one students, twenty one students remained and. with 
the facilitator, formed a community of learners. These students were equivalent to thirty four percent of the 
population (21 divided by 61 equals 34.4262%). Finally, sixteen students passed the course. Fourteen 
students submitted their learning journeys. Many included mind maps. About half got an A+ or an A. 

The 34% outcome was consistent to the research of Brown and Venkatesh (2003), arguing that sixteen 
percent of the one hundred percent population are initially positive in favor of the innovation (e.g., the 
authentic learning) and forty percent of the eighty four percent (that is, 40% * 84% = 33.6%) against it could 
be persuaded to accept the innovation in the end. In short, the people of the population who would accept an 
innovation range from a minimum of 16% to a maximum of 49.6%. The twenty one students who accepted 
authentic learning (the 34%) belonged to the mentioned segment. 
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4.2 Authentic Learning as an Agent of Change 

Through authentic learning curriculum, Kazakhstani students evolved from passive learners to active 
learners. Student #1 recalled her learning journey, which started with hard knowledge in the textbook and 
ended with soft knowledge. 

“Generally, the topics we were covering grew more complicated throughout the course, as they 
became closer to reality. Indeed, in practice, things are more complicated and not as easily 
approachable as described in the textbook. In reality, each company has its own unique features 
and performance patterns that need certain adjustments. These adjustments involve more judgment 
and soft knowledge.” - Student #1 

Likewise, students who conducted actively with their independent in-depth learning engagements realized 
a heightened cognitive presence. An evidence is the recognition of unique features and patterns (see previous 
quote). Another evidence is the construction of knowledge in stages. According to Student #1, the first hard- 
pure stage established the foundation. In this stage, the lectures were entirely delivered by the facilitator. The 
second hard-applied stage included simple and straightforward application of theory in practice. The 
knowledge was based on textbook examples. Although the lecturer maintained the leading role, the learning 
process gradually involved interactions with other students in small groups. 

“[The] third ... soft-applied stage... was the longest and important stage associated with most 
advanced real-life approach to the study. It was aimed to apply the previous experience and 
knowledge to existing companies in a more sophisticated real-life environment. It used more 
complicated examples... [The] fourth ... soft-pure stage... had an essay worth 20% of the grade 
that had to cover the practical issues and problems we face[d] while implementing the final project 
and others. These problems had to be considered not merely in the frames of the particular project, 
but rather in relation to the whole country’s financial system. It was significant to realize how 
these problems affect the decision quality made by investors not just in the case of the final 
project, but. ..[with] regard to all companies in Kazakhstan. . ..[Our] understanding of financial 
analysis expanded to the scope of the environment and society we lived in.” - Student #1 
Student #3 likewise experienced heightened cognitive presence as well as social presence. He recognized 
the sensitive factors of capital structure of the firm, realized the privilege of being empowered to choose and 
make decision, and the effective learning. 

“[W]e decided to take a particular Kazakhstan company, and calculate all of the ratios for it. . . We 
also decided to make our presentation more interactive... [and] to encourage students to be more 
active... I understood more about the risk of the company and how it depends on the company's 
capital structure. . . (italics emphasized)” - Student #3 

Heightened social presence accompanied the heightened cognitive presence. Students recognized the 
motivation to solve problems as a community of learners. 

“The process of working on the bonus projects was amazing... [W]e all were working in the 
groups, communicating, observing and looking for the solution together. We all received 
support... from each other. ... It was a terrific motivational tool for making everyone engaged in 
the process, discussions and conversations. The chance to give the lectures to other classmates 
makes you responsible for the information that you give to others, as you should understand 
everything concerning the topic and should be prepared to explain it in more detail. [The 
discussion, a] winning strategy in [learning, making] the class more volatile in reaction to the 
learning activities and new topics and are consistent with the studying procedure.” - Student #4 
A key feature of authentic learning is the uncertainty. Student #4 anticipated the uncertainty (e.g., “more 
volatile in reaction”). Likewise, Student #5 experienced the same feeling. 

“The whole course for me was like [a] jump. When I got those yellow cards and all supplementary 
materials, it seemed to be clear how, what and when to do the activities. . . However, soon you are 
totally disoriented and don't know what to do. There is so much uncertainty and you don't know 
how to start. . . [Teaching and learning] ease the educational process, as there is no need to research 
every question; some of them were already done by other students. At some point, I realized that 
learning became much easier. Finally, you see the result of an excellent job and realize that it was 
not as difficult as it seemed to be at some point.” - Student #5 
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4.3 Authentic Learning as Knowledge Development Stages 

The four stages of learning, as conveyed by Student #1, actually correspond to the four types of knowledge, 
namely: the hard-pure knowledge learned from the lecturers; the hard-applied knowledge learned from 
cooperative learning and problem solving; the soft-applied knowledge, requiring intensive use of 
spreadsheets, advanced real-life approaches, and a high level of judgment; and finally the soft-pure 
knowledge where students took everything they learned and wove them into their new knowledge. Student #1 
understood that the knowledge covered both the classroom issues and the wider context of society. 

“Presentations of our co-learners demonstrated, using the example of existing companies, how to 
apply the theoretical materials introduced in Chapter 13. This involved an Excel spreadsheet 
application... to enhance our learning experience and develop our practical skills. [The] 
spreadsheet became the main part of every presentation ... to explain [the] practical side of every 
topic... Chapter 14 was introduced to widen our learning of broader, abstract and soft topics like 
accounting quality and forecasting... Additional bonus projects were undertaken to enrich our 
practical experience and go beyond textbook materials. These projects focused on assessing the 
risks of existing start-up companies. The objectives of the project were to learn how to obtain 
necessary information from online data directories and how to use historical information and 
statistics... It is significant to mention that softer skills and professional judgment were used 
during this task.” - Student #1 

The construction of knowledge did not happen only through group projects. The reflection on the learning 
process was imposed as a means to construct knowledge. However, not many students (like Student #2 
below), realized the importance of the thinking about the learning process. 

“I didn't like reflections; they took so much time... But work on projects was lik e fresh air for me. 

I liked them most, because we tried to be creative, innovative and tried to implement our 
knowledge from all fields. I wish we had more [of] that kind of work... [I also] liked that all our 
works were evaluated with comments, (italics emphasized)” - Student #2 

Heightened cognitive and social presence also came from the focus-group discussions at the start and 
toward the end of the semester, the assignments, and the summative learning journey. This heightened 
presence, in turn, was essential to quality knowledge development stages. 

4.4 Authentic Learning as Constructed Knowledge 

The knowledge development stages underscore the knowledge constructed, which comes in the form of 
communal artefacts (projects and situated works) and which is created when teaching and learning methods 
connect the individual learners to the community of learners. Student #2 conceived the project as a breath of 
fresh air, going beyond textbook materials, enriching practical experience, encompassing knowledge from 
many fields, and empowering learners to be more creative and innovative. The emphasis is on collective and 
non-isolated knowledge structures. For instance. Chapters 13 and 14 were read not in isolation from other 
chapters that were also critical to the understanding of the valuation of the firm. The sense of connectedness 
and integration is the heart of constructed knowledge. 

4.5 Authentic Learning as Empowerment 

Constructed knowledge in authentic learning puts greater emphasis on collaborative inquiry and on the 
opportunities for students to make choices and create the epistemic artefact (e.g., the project). Students 
realized that collaboration and empowerment to make the learning easier (refer to the reflections of Student 
#3, Student #4, and Student #5). 

The case study revealed that providing choices to students results in better learning because they see the 
learning as context-rich, diversified, and relevant to their needs and motivations. The empowerment is not 
limited to what is learned, but includes the environments (schools or classrooms), situations (formal or 
informal), and other traditional or non-traditional (radical) alternatives. Yet there do remain unavoidable 
mandates, such as, in this case, the focus on Chapter 13 and 14 and on weekly reflection on the learning 
process. 
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5. DISCUSSION 

The case study has validated the four truth statements corresponding to the four issues in Figure 1. Firstly, 
authentic learning acts as an agent of change, helping students develop positive perspectives about the 
realities affecting them. Unlike traditional learning, authentic learning demands heightened cognitive 
presence, heightened social presence, and heightened (teaching and) learning methods. Such a triple 
heightened presence in turns requires (1) a community of learners, (2) the credibility of the facilitator and 
students, (3) the innovativeness of the facilitator, (4) the commitment of students to independent in-depth 
learning, (5) positive attitudes and beliefs of co-teachers and co-learners, and (6) a positive and empowered 
learning environment. All these elements make authentic learning curriculum design an art in development, 
facilitation and engagement. 



Figure 1. Argumentation 

Back to the corollary question: What patterns did the students conceive in their learning? Student #5’s 
description of authentic learning as a jump (a force) acknowledge its elements of instability and uncertainty. 
Student #1 expressed authentic learning as a means (a journey) to develop one’s codified (hard) 
understanding of financial analysis first into judgmental (soft) understanding and later into something even 
broader and more abstract. 

The researcher found that many students wanted to be challenged by authentic learning. That is, they 
were able to see the value of trying to solve the real-world problems with a community of co-learners. Thus, 
authentic learning is an agent of change worth contemplating. 

The learning journeys examined reveal the readiness of students to commit to authentic learning (the first 
assumption in Table 1) and assume that the same commitment was not present in the students who withdrew 
from the course. But two students did not submit their learning journeys and could be presumed to have 
lower readiness, although they nevertheless passed the course. The fourth segment could be students who had 
the readiness but chose to withdraw due to peer pressure. The implied challenge is to find ways to reduce 
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negative bias so that authentic learning appears effective and meaningful. 

A second assumption is that authentic learning cannot happen apart from the contexts and must be 
modified to fit the individuals, the small groups, the whole class, the course, and the institute of higher 
education. For instance, an Introductory Accounting for freshmen would require a different design from the 
FSA course. 

A third assumption presumes that students learn better if the learning is relevant to their needs or 
interests. This takes for granted that students know what they want. Yet there is always the possibility that 
they might choose something that would not maximize their learning. The curriculum designer’s concern 
becomes how to minimize this risk. 


6. CONCLUSION 

From the segment of students who remained in the section, the evidence shows that students recognized the 
alchemy in the course (Issue #1 in Figure 1). The authentic learning as an agent of change requires 
heightened cognitive presence and heightened social presence to be able for the co-teachers and co-learners 
to create positive perspectives from the micro level of the learners to the macro level of society. 

Second, students evolved from passive learners to active learners through several stages. Aside from the 
heightened cognitive presence and heightened social presence, a third pre-requisite in the knowledge 
development stages is the heightened teaching and learning methods, such as true inquiry, thinking, thinking 
about thinking (meta-cognition), and discourse (Issue #2 in Figure 1). 

Third, the construction of knowledge connected the in-here judgment with the out- there performance 
patterns of the firm. At the same time, the learning engagements were means to an end. namely, the 
summative assessments. Students appeared to recognize both the deterministic and instrumental perspectives. 
Thus, the constructed knowledge is the outcome of the authentic learning process. It highlights the learning 
environment and the teaching and learning process and experience. 

Fourth, an essence of authentic learning is empowerment, which comes with providing students with 
adequate opportunities for them to make choices and to design their learning pathways, challenges, and 
experiences. 

The co-teachers and co-learners in a community of learners need to have heightened awareness of these 
elements. The study affirms that only teacher who recognizes authentic learning would design and implement 
such a non-traditional curriculum, while only students with a positive biased toward authentic learning would 
remain and actively engage in the course. Furthermore, the study confirms that many senior Kazakhstani 
students have the readiness and the desire to be challenged by authentic learning. They see themselves 
becoming active learners in a nurturing authentic learning environment, and later on, experts in their chosen 
fields. 


7. DIRECTIONS FOR FUTURE RESEARCH 

Authentic learning can potentially cross -pollinate learning opportunities across disciplines (e.g., accounting, 
marketing and entrepreneurship) or across settings (e.g., learners belonging to multi-cultures). It also needs a 
different balance of informal and formal learning opportunities as people move across multiple life-spaces. It 
would be useful to look at how, when, and where learning occurs for people who straddle multiple life- 
spaces. 

Authentic learning holds students responsible for conducting independent in-depth engagements and also 
holds the facilitators responsible for creating and nurturing a harmonious learning environment. These two 
assumptions, however, do not apply to students who refused to commit themselves. The future research could 
explore ways to capture the reasons for withdrawing from the course. Lastly, this exploratory study is of 
instrumental interest to positivist research as an agenda for future research. 
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ABSTRACT 

This paper presents results from a multidisciplinary program targeted at engineering education and at the development of 
entrepreneurial mind in telecommunications engineering students. 

The basic concept is rooted in a capstone-like project with the following characteristics: (i) Creation of student awareness 
about real world engineering activities, involving engineering professionals and enterprises in selected classroom 
activities; (ii) Simulation of a business environment in capstone project classes; (iii) Market gaming around a set of 
business cases where students are organized in teams playing different professional roles (role playing); (iv) Linking the 
outcome of market gaming and associated business cases with syllabus topics and with practical issues resorting to 
engineering decisions that have to prepared based on technology choices, network design, market simulation and 
economic-financial analysis. 

The results showed an improvement in average marks, the development of teamwork skills, the consolidation of previous 
knowledge and a better understanding of the telecommunications business markets. 


KEYWORDS 

Tutoring, problem based learning, entrepreneurship. 


1. INTRODUCTION 

The profound mutations that took place over the last decades in terms of enabling technologies, emerging 
business models and organizational structures challenges continuously the telecommunications sector and 
demands, more than ever, a competent and flexible workforce. The sector’s companies look for graduated 
engineers equipped with the competences required by the new professional environment. However, a well 
succeeded recruitment is hard to achieve, as recent graduated engineers are considered not sufficiently 
prepared with knowledge and skills to face the uncertainties of the market. They frequently lack an integrated 
vision of the telecommunications sector, sound scientific background and ability to cope with technological 
and organizational changes. Some fundamental soft skills such as problem-solving, teamwork and 
communication have also been referred by companies’ as increasingly important. 

Higher Education institutions have responsibilities in preparing future engineers to work competently and 
deal with unpredictable challenges. Unfortunately, engineering curricula are still characterized by a strong 
emphasis on science and technology disciplines instead of more practical approaches. Science and 
technology are essential requirements to prepare students with the analytical skills that an engineer must 
have. However this preparation on propaedeutic and specific subject matters of each engineering field 
frequently is not accompanied by an effort to prepare students about equally important non-technical aspects 
of their profession. These shortcomings are particularly felt in relation to soft skills such as planning, 
organization and inter-personal communication. All this is further aggravated when they have to work within 
a team. In addition, it is also frequent that during their courses, students develop very little awareness about 
the outside world, namely about the markets where soon they will be looking for a job or fighting to keep it. 
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For many engineering graduates, when starting a career, the unsuitability of companies’ demands and 
educational programs results in serious behavioral mismatches and very limited knowledge about the activity 
sectors and businesses where they become involved These circumstances can represent an important 
handicap in their careers and the resulting limitations can significantly impair their capability to play the roles 
that enterprises expect from them. In addition, these weaknesses also do not favor the emergence of an 
entrepreneurial spirit [1] among young engineers, restricting their ability to contribute to economical and 
social growth. Ultimately, all this can jeopardize their employability. 

This situation creates new responsibilities on the part of Higher Education institutions. Curricula should 
match industry needs, not only in terms of contents but also in terms of pedagogical approach. Learning 
processes should be focused on the learner, captivating his interest and promoting active and autonomous 
competence development. 


2. EXPLORATORY STUDY 

Over the last 8 years a study has been conducted encompassing approximately 250 students of engineering 
courses (higher education) and approximately 500 students of foundation courses (post-secondary education) 
[2]., in other to gain a better understanding about the matching between industry needs and curricula, and 
also to prepare future actions. This study addressed the following aspects: 

• Student’s representations with respect to the specific subjects of study of their courses. 

• Representations of enterprises that received either young graduates or trainees from engineering and 
foundation courses. 

Among the findings of this study where the following aspects: 

• Engineering and technology students receive tools for solving problems that they have never faced 
before and for which they do not have an adequate appreciation. 

• Because of their limited real world experience, engineering and technology students have difficulty 
in understanding the practical applications of their studies. 

Another frequent feeling among large amounts of engineering students is that they find that classes 
were boring [3]. Previous research [2] found that this is mainly due to the following causes: 

• First, because of their limited real world experience, students have difficulty in understanding the 
practical applications of their studies; 

• Second, as a direct consequence of the traditional universities’ teaching approach: students receive 
tools for solving problems that they have never faced before and for which they do not have an 
adequate appreciation. 

In summary, many students don’t develop meaningful knowledge and competences and this is caused by 
the adopted pedagogical approaches that don’t promote active learning. 


3. ROLE-PLAYING: PROMOTING ACTIVE AND MEANINGFUL 
KNOWLEDGE 

In order to provide answers to the problems identified in the study outlined above, a pedagogical initiative 
was launched targeted at the promotion of active and meaningful knowledge creation in engineering students. 
This initiative is currently taking place in the context of several courses in the area of Electrical Engineering 
(with majors in Telecommunications and Information Systems) at post-secondary, B. Sc. and M. Sc. levels 
(Bologna system). The basic concept behind it is rooted on a capstone-like project where groups of students 
play the role of telecommunication companies competing against each other resorting to decisions that they 
have to prepare based on sound engineering studies: technology choices, network design and dimensioning, 
market simulation and economic-financial analysis. 

With this role-playing approach students are engaged in authentic real-world problems and active 
learning, having the opportunity of learning by doing, receiving feedback, continually refining their 
understanding and building new knowledge [4], This initiative is intrinsically dynamic, learner-centred and 
more experiential than traditional ones. It represents an attempt to improve student's classroom involvement. 
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bridging the gap between the engineering profession and the classroom, attempting to contribute towards 
better success rates and improved employability. It also helps the development of professional identities. 

The initiative followed two phases: 

• First, definition of project ideas made with the contributions of practicing engineers from several 
companies that are invited to present some of their real-work challenges in a series of seminars. 
Students engaged in weekly discussion sessions with practicing engineers and experts (industrial 
guest speakers) in order to exchange ideas and discuss career paths. The main objectives of these 
sessions were: provision of the “big picture” about core characteristics of what telecommunication 
engineers do; exposition to positive role models; encourage questions and understanding. 

• Second, projects designed around a situation where teams play the role of competing companies in a 
market place. Competition initiatives among teams playing the roles of competing companies in an 
open market were delivered, in order to expose students to business dynamics [5], 

This leads to an atmosphere of project-based active learning combined with an interactive entrepreneurial 
atmosphere in the area of telecommunications engineering. The role-playing competitions followed three 
steps: 

• Faced with a specific challenge (as will be outlined ahead in the paper) each team tries to identify 
possible solutions and must make its evaluation, both in technical and economical terms. 

• Chosen solutions must be converted into a business case, with different teams playing the roles of 
competing companies in a marketplace. 

• A didactic market simulator is used to create condition similar to those found in real markets and to 
convey experimental lessons transferable to the real world. 

A description of this market simulator is provided next. 


4. DIDACTIC MARKET SIMULATOR 

Training simulators are designed for education purposes, providing significant hands-on experiences that 
motivate and facilitate learning [6], Additionally they can also offer experiences that resemble those of the 
real world and, thus, can give students the opportunity to apply theory in an efficient, economic and 
interactive fashion. 

The work described in this paper was supported by the usage of a didactic market simulator that can be 
used to make students familiar with the dynamics of the telecommunications sector. It can easily be 
transposed to other economic sectors. Its structure, in its present state, is depicted in the following figure: 


Input Decisions 
Actor 1 j 

Actor 2 ~ 


Actor n 


Outcomes/ 



Engineering options: 

- Network Architecture 

-CAPEX 

-OPEX 

- Dimensioning 

- Roll-out planning 
-etc 


a Market Parameters 


Market Outcomes: 

- Market shares 
-Cashflows 

- NPV 
-HR 
-etc 


Commercial options: 

- Package of Services 

-Tariffs 

-etc 


Figure 1 . Didactic market simulator structure 
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4.1 Purpose of the Didactic Market Simulator 

The simulator is designed so that students will learn how telecommunications engineering decisions (e.g., 
network architecture, physical media, bandwidth, latency) associated with marketing, economic and financial 
decisions (e.g., offered services, tariffs, competition among operators, etc) affect the overall network 
performance and the ways markets react. 

In a preliminary phase Excel was used as the basic platform. This enabled some fine tuning of the 
mathematical model and also proved very useful for the determination of several parameters. At a later stage 
the implementation was migrated to a web environment supported by database and appropriate query 
languages. 

The following figures illustrate some screen shots obtained with the market simulator in a web 
environment: 


Player Home 
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Current Tick UPEX 

Current Net Present VMue 
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Current Tick Revenues 16.069.524.92 C 

Current Tick Investment 6.978.301 .47 € 


Current Intemii Rate of Retur 


Current Simulation data 



Current Market Share 



■ ■ OPtX ■ -*• fUUrvt* CAFfX 

Figure 2. Example of screen shot obtained with the market simulator (investment analysis in an access network with 3 

operators) 


4.2 Test and Validation 

In order to test and validate the approach described in this paper a series of experiments was trialed over the 
last two years. This was done in the context of a Capstone Project in the 3 ld year of an MSc in Electronics 
and Telecommunications Engineering (total duration: 5 years ; 3 years 1 st cycle; 2 years 2 nd cycle). 

The basic objective of this Capstone Project is to face students with the challenge of projecting an access 
network using up-to-date technologies (e.g., Fiber-to-the-home, LTE, WiMAX, etc) and evaluating the 
different architectures (point-to-point, point-to-multipoint, etc), different engineering solutions (active, 
passive, etc), roll-out strategies (market size estimates, time plan of investments, tariffs, etc). In this work 
students are required to integrate knowledge and skills developed in other disciplines, probably over a period 
of more several years. 

To estimate (quantitatively) the impact of the approach described in this project on student learning and 
understanding, during the last 3 weeks of a semester (over the last 2 academic years) , the class (45 students, 
average) was given an assessment test (multiple-choice questions) on Access Networks (a subject not 
specifically studied in the 9 preceding weeks, and which requires the integration of knowledge and skills 
developed in other disciplines, over a period of approximately two years before the capstone project) and the 
market simulator was introduced. 

After this test the class had the opportunity to attend a seminar (1 hour) by an invited senior 
telecommunications engineer responsible for the access network planning in a major telecom operator. Here 
students had the opportunity to witness some of the challenges faced by a telecommunications engineer in 
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planning, designing and operating an access network under severe market competition conditions. At this 
point the class was split in 9 groups of 5 students for a short period ( 1 hour) doing hands-on familiarization 
with the market simulator. This was followed by a period of 2 more working sessions (4 hours over 2 weeks) 
where the class was organized in sets of 3 groups. In each set each group played the role of a telecom 
operator competing with the other. 

Tablet lap-tops were made available for these sessions in order to facilitate interaction and discussion of 
ideas inside groups and among groups. 

In the first of these 2 sessions every group started with equal market share as the other groups. Following 
a choice of engineering options related to the specific access network under consideration (architectures, 
active or passive network elements, market size estimates, expected competition, time plan of investments, 
tariffs, etc,) the simulator produced the market share situation for every competitor, corresponding to half of 
the study period under consideration (as illustrated in Erro! A origem da referenda nao foi encontrada.). 
During the period until the following session, in the week after, every group tried to devise possible strategies 
to either recover from the bad position where the first run had left them or to keep the advantage that 
eventually they had already obtained. The second run dictated the final results of the market game. 

After this experience an assessment test on Access Networks as similar as possible to the original one (but 
not equal...) was given again to all 45 students in order to measure eventual changes in student learning and 
comprehension. 

Figure 3 shows the aggregate results of these tests over the period of 2 academic years. 



0 5 10 15 20 

♦ Change in Score Poly. (Change in Score) 


Figure 3. Impact of the capstone project approach on student outcomes 

4.3 Assessment of Experience’s Impact on Student’s Learning 

The results obtained, in spite of referring to just 2 runs of the experiment over the last 2 years (other will 
follow in subsequent years), were very encouraging: 

• The classes, as a whole (2009-9, 2009-10) showed an average improvement of 2,06 points (out of 
possible 20), that is, approximately 10,3%. 

• It was interesting to notice that the improvement was particularly significant in students with 
average marks, where the vast majority of engineering students do stand more frequently. 

The above results were complemented by a set of (informal) interviews with a sample of 10 students 
(out of 45), in both academic years, in order to gain some feedback about how students felt with the 
experiment. The outcome of these interviews was generally very positive, underlining particularly 
the following aspects: 

• The very positive effect of having a practicing engineer sharing with students some of it 
professional experience in problems very similar to those that they were facing in the capstone 
project (a typical case of “situated learning” [3]). 

• Having the possibility to play with the didactic market simulator proved to be extremely useful to 
integrate and consolidate previous learning, to help gaining a better understanding of businesses 
dynamics and to improve teamwork. 
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4.4 Assessment of Experience’s Impact on Employers 

Given the fact that the experiment was done with students attending their last year of the 1st cycle of the 
engineering degree (Bologna type) it was possible to track some of these students in their first employment. 
This was done with a group of 5 students that graduated in 2008-9. Results and is currently being done with 5 
additional students that graduated in 2009-10. 

As part of this exercise several interviews were made with responsible personnel of the employing 
companies following the first 3 months of employment of the graduates. 

In spite of the limited statistical value that this limited number of enquires might have it is very 
encouraging to notice that, in general they seem to point out to the following: as compared to their company 
colleagues, test graduates exhibit better teamwork skills, show good ability to integrate and associate 
knowledge from different fields and reveal good understanding of the telecommunications business markets. 


5. CONCLUSIONS AND FUTURE DEVELOPMENTS 

The results obtained, in spite of referring to just one single run of the experiment (other will follow in 
subsequent years), were very encouraging: 

• The classes as a whole (2009-9, 2009-10) showed an average improvement of 2,06 points (out of 
possible 20), that is, approximately 10,3%. 

• It was interesting to notice that the improvement was particularly significant in students with 
average marks, where the vast majority of engineering students do stand more frequently. 

• The above results were complemented by a set of (informal) interviews with a sample of 20 students 
(out of 90) in order to gain some feedback about how students felt with the experiment. The 
outcome of these interviews was generally very positive, stressing in particular the following 
aspects: 

• The very positive effect of having a practicing engineers sharing with students some of their 
professional experience in problems very similar to those that they were facing in the capstone 
project (a typical case of “situated learning” [5]). 

• Having the possibility to play with a didactic market simulator closely linked to the engineering 
variables present in a typical telecommunication project proved to be extremely useful to integrate 
and consolidate previous learning, to help gaining a better understanding about businesses dynamics 
and to improve teamwork skills. 

• As compared to their company colleagues, test graduates exhibit better teamwork skills, show good 
ability to integrate and associate knowledge from different fields and reveal good understanding of 
the telecommunications business markets. 

The implementation of role-playing activities proved to be a fruitful pedagogical technique with the 
potential to transform theoretical concepts into experiential outcomes. In this way, educational role-plays 
engage students in close to real-world learning, providing students opportunities for learning by doing, 
refining their understanding and building new knowledge. Improved employability is also a potential 
important result since graduates capability to play the roles that enterprises expect from them is significantly 
benefited. 
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ABSTRACT 

In the education sector, teamwork assessment and self/peer evaluation are widely applied in higher education nationally 
and internationally. This assessment is designed to encourage students to promote and improve their skills in teamwork, 
communication (writing, interpersonal interaction and cultural awareness, and presenting), critical and creative thinking 
(problem solving and decision-making). Information Technology literacy, and information literacy. Beside the above, this 
assessment will improve work performance and productivity and increase self-confidence in a range of situations. These 
skills are developed by formative and summative feedback, collaboration, and cooperation between students and the 
lecturer. This study aims to discuss the development of teamwork assessment and self/peer evaluation in the BT2 
undergraduate unit at an Australian university to promote students’ skills and to prepare them for the workplace in the 
future, as most of these skills are indispensable not only from the academic perspective but for business as well. This 
study provides empirical evidence from 267 students, based on quantitative and qualitative data derived from two 
sources. The first is the anonymous informal feedback collected during the semester, while the second (formal) source of 
students’ evaluations and perspectives towards the BT2 unit is ‘eVALUate’, the anonymous online system for gathering 
and reporting students' perceptions of their learning experiences at the university. The students’ remarks show their 
satisfaction with this assessment, as it develops specific skills for the current study and for the future workplace, i.e. time 
management; problem solving, decision-making, cultural awareness, presentation, communication and meeting a 
deadline. 


KEYWORDS 

Teamwork assessment, self/peer evaluation, learning skills, higher education, Australia 


1. INTRODUCTION 

From the business perspective, it has been noted that Information Systems practitioners have been eagerly 
exploring the need for effective skills, i.e. teamwork, critical thinking, writing, communication, and oral and 
written presentation, among Information Systems graduates (Lanning et al. 2011; Van Deursen & Van Dijk 
2009). Thus, university academics have started to integrate these skills in their curricula and units to assist 
students in university life, their future workplace and life in general. To align with businesses’ needs and 
desires, a teamwork assessment and self/peer evaluation was developed and integrated in the Business 
Technology (BT2) unit to motivate and challenge students to explore and investigate the strengths and 
weaknesses of the new technologies in the marketplace and present the outcomes as a report in an academic 
style. Furthermore, this assessment will identify how these technologies will assist businesses to improve 
their performance, increase productivity, and raise their awareness and variation in the workplace. The 
teamwork assessment discussed innovative topics, namely ‘fiber to the home’ (FTTH); security, the Internet, 
and the latest operating systems; mobile technologies and Internet telephony; 2015 new laptops; wireless 
technology, i.e. WiMax (Worldwide Interoperability for Microwave Access); green technology; cloud 
computing; IPv6; and virtualization. Teamwork assessment and self/peer evaluation are becoming an 
essential part of BT2 unit, to encourage students to learn, improve and develop communication skills (such as 
writing and oral presentation), to develop excellent relationships with their peers, and to exchange ideas and 
knowledge in relation to the unit in general and the assignment in particular. The teamwork assessment and 
self/peer evaluation were released in week one of the semester, with the necessary information from the 
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marking guide, strategies on the presentation and structure of reports, and guidelines on plagiarism and 
referencing style. The self/peer evaluation aims to encourage each student to think about and understand how 
well s/he and her/his partner performed the team activity, and to develop their teamwork skills. Each pair 
submitted one peer-reflective and one personal (self-reflective) comment on what s/he has done for this 
assignment, and a copy of her/his notes on the resources that s/he developed or located in this assignment. 
The self/peer reflection questions encompassed the following headings: (1) Individual roles and 
responsibilities; (2) Generating and creating ideas/strategies; (3) Research; (4) Collegiality; (5) 
Organizational skills; (6) Final outcome/product. 

Furthermore, students were asked to complete the self/peer reflection as well as the peer reflection 
comments section. The former included three questions or tasks - (1) what did s/he leant from this group 
activity (in terms of the process of working in teams - not content/information)? (2) What would s/he need to 
change in the way s/he worked with her/his partner in order to make future team activities more successful? 
(3) List three strengths that you have in team working and two weaknesses that you need to continue to 
improve, while the latter included one question that asked the student to list three strengths and two 
weaknesses that s/he had identified in her/his partner from this team-working assignment. Each student 
uploaded their evaluation and draft resource material to the blackboard FLECS facility, and later the lecturer 
checked and assessed the group’s work and identified how well the pair functioned. Finally, this assessment 
provided the students with an opportunity to inform the lecturer if there were any problems with the pair 
interactions, the division of workload, or other aspects. On the basis of this information, the lecturer had the 
capacity to assign each student a different mark based on his or her efforts to complete the assignment well. 
This paper will present an Australian perspective on integrating teamwork assessment and self/peer 
evaluation in higher education. This study is organized as follows: (1) teamwork assessment and self/peer 
evaluation; (2) methodology and research question(s); (3) BT2 Participations, (4) BT2 Assessments, (5) 
Results, (6) Discussion and Lessons Learned, (7) Conclusion. 


2. TEAMWORK ASSESSMENT AND SELF/PEER EVALUATION 

Teamwork assessment and self/peer evaluation have become widely used in university courses, including 
those on computing and information systems (Lejk, Wyvill & Farrow 1996). It has been noted that the 
integration of teamwork assessment and self/peer evaluation in the higher education curriculum would 
develop skills in communication, leadership, time management, problem solving and decision making 
(Gielen, Dochy & Onghena 2011; Oldfield & MacAlpine 1995; Weaver & Esposto 2011). Furthermore, this 
assessment will provide more interesting, effective, and exciting learning experiences for students and the 
lecturer, as plenty of new ideas and perspective will be generated, and more discussion and debate will occur 
(Lanning et al. 2011; Nulty 2011; Weaver & Esposto 2011; Webb 1995). Furthermore, the literature review 
confirmed (Diggins 2004; Gillies & Ashman 2003; Lanning et al. 2011; Pavolvich, Collins & Jones 2009; 
Steensels et al. 2006) that students who have completed assessments in teamwork mode would be able to 
work collaboratively with others in the university and in the workplace in future; the significant concept 
behind this assessment is to develop and promote students’ skills for the lifelong learning process. 

BT2 is a core unit in the double/single major in the Bachelor of Commerce degree (information systems 
and information technology, information technology and digital design, information systems and internet 
communications, information systems, and information technology). The BT2 unit focuses on hardware, 
software, operating systems, networking, LAN protocols, network services, and UNIX scripting. According 
to the university policy (Curtin University n.d.), each unit must promote two to three professional skills: i.e. 
communication (writing, interpersonal interaction and cultural awareness, and presenting), critical and 
creative thinking (problem solving and decision making), teamwork, IT literacy and information literacy. The 
BT2 unit promotes the following professional skills: (1) Critical and creative thinking - problem solving; (2) 
Critical and creative thinking - decision-making; and (3) Teamwork. Integrating these skills in BT2 assists 
students to (1) identify problems and analyze the main features; (2) identify, implement and evaluate 
strategies for the resolution of problems; (3) generate pioneering solutions; (4) accept each other’s ideas 
within the group and gather all these ideas to generate a logical argument to solve the problem of the 
assignment; (5) manage the time effectively, which involves meeting requirements set by others; (6) manage 
projects effectively, which involves the organization and co-ordination of group work; finally, (7) negotiate 
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successfully with others, which involves getting people to move from a position they prefer to a position that 
suits all parties. As for the teamwork assessment and self/peer evaluation in BT2, it allows students to 
develop the following skills: (1) communication, information literacy, information technology literacy and 
teamwork skills, (2) evaluating how teamwork is beneficial in distributing the workload (ensuring individual 
accountability), creating participation and involvement, making better decisions and generating a diversity of 
ideas. To allow the smooth development of these skills amongst the students, an icebreaker technique was 
adopted in the lab session in which the students were asked to introduce themselves to the lecturer and the 
other students simultaneously. The icebreaker questions included name, degree, current occupation (if any), 
experience in computers, and education in computers; ‘Why did you select this unit?’, and ‘What do you 
expect to gain from this unit?’ This technique was very useful in the lab session for allowing students to gain 
more information about their peers with respect to studies, knowledge and work experience. In week two, the 
students chose their peer for the team working assessment from the same lab cohort, to monitor and check 
their progress for the assessment. Furthermore, the procedures for conducting this assessment were explained 
in the lab session and in the lecture with respect to the report writing task, plagiarism and referencing. Also, 
to emphasize the importance of these procedures, a presenter from the communications skill center in the 
university was invited to emphasize the importance of report writing, plagiarism and referencing, as a useful 
handout showing different examples with respect to the above procedures was presented to the students. 
Students were monitored during the whole process, and if any conflicts occurred between team members in 
relation to teamwork assessment, or if students did not appear to be getting along, a contact would be 
established with students in order to assist them to finalize and submit the teamwork assessment and the peer- 
and self-reflection on time. Furthermore, after the assessment was submitted the self/peer evaluations were 
checked and compared to decide if the teamwork contract between the pair was workable. Nevertheless, if a 
major difference occurred in the self/peer evaluation, then students would be contacted to clarify the situation 
and later each student would receive a different mark for the assessment. 

Furthermore, based on students' formal and informal feedback, slight changes were carried out to the 
teamwork assessment and self/peer evaluation. These changes included allocating marks to the self/peer 
evaluation to emphasize the value of this assessment, and two approaches were introduced and integrated into 
the teamwork assessment and self/peer evaluation: (1) student expectation process; (2) teamwork agreement 
and timeline. The former was intended to assist students to choose partners who had similar expectations to 
themselves and who were aiming for similar grades for the group assignment, as the icebreaker technique 
became unpopular in the lab session. The latter assessment was introduced since in previous semesters many 
students had trouble in finalizing the assessment because of the deadline. To prevent this problem, a 
teamwork agreement and timeline were signed by the pair and submitted to the lecturer during week three of 
the semester. During the lab session, the lecturer tracked the students’ progress on the basis of the teamwork 
agreement, as students identified which task activities should be completed in each week. The lecturer 
provided formative feedback to tackle any problems immediately and to improve the presentation, structure, 
and layout of the assessment. Some outstanding feedback was received from the students regarding the recent 
changes, as the majority of the students agreed that these approaches allowed students to manage their time 
efficiently and to complete the assessment on time. Finally, more comments from the students indicated that 
the teamwork agreement and timeline allow them to: (1) finalize the assessment promptly, and achieve better 
planning and preparation, (2) learn new skills and knowledge from the other group members, (3) split the 
workload effectively according to the agreement and (4) organize the assessment work and reduce time 
pres stue. 


3. METHODOLOGY AND RESEARCH QUESTION 

The mixed methods approach (quantitative and qualitative methods) was used in this study to minimize 
discrepancies in the findings and provide a substantial amount of data to identify the positive and negative 
aspects of the adoption of teamwork assessment and self/peer evaluation in the BT2 unit (Creswell 2003; 
Crump & Logan 2008; Gilbert 2006; Harrison & Reilly 2011; Hesse-Biber 2010; Maudsley 2011; 
Molina-Azorin 2011; Teddlie & Tashakkori 2009). Currently most researchers and scholars are in favor of 
mixed methods in a single research study, since this mixture allows researchers to have a comprehensive 
understanding of the data collected through their research, by a combined analysis of text and statistical data. 
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Creswell (2003) proposes that mixed methods are an approach for collecting, analyzing and integrating both 
quantitative and qualitative data in a single study to understand the problems and to identify the strengths and 
weaknesses behind the study. By the same token, Wiggins (2011, p.47) claims that “the strengths of each 
methodology (flexibility and intimacy for qualitative, generalizability for quantitative) make up for the 
other’s weaknesses (low generalizability for qualitative and rigidity and objectivity for quantitative)”. In this 
study, the data were collected using quantitative and qualitative methods via anonymous students' formal and 
informal feedback. Informal feedback was collected during the fourth week of the semester. The informal 
feedback was designed to give the lecturer some idea of how students felt about the unit and learning 
experiences so far in the semester. The informal feedback was analyzed and shared with the students in order 
to carry out the changes before the end of the semester. Questions in the informal feedback included: What 
are the positive aspects of this unit? What do you like about the lecturer? Do you have any suggestions or 
recommendations for how the learning experience can be improved? (Please provide specific suggestions.) 
What do you need the lecturer to continue doing, to stop doing, and to start doing? As for formal feedback, 
‘eVALUate’ is an anonymous university feedback system that usually runs at the end of the semester. 

This feedback mainly focuses on the unit as well as satisfaction with the teaching. The formal feedback 
included responses to the following statements: the learning outcomes in this unit are clearly identified; the 
learning experience in this unit helps me to achieve the learning outcomes; the learning resources in this unit 
help me to achieve the learning outcomes; the assessment tasks in this unit evaluate my achievement of the 
learning outcomes. Moreover, the feedback on my work in this unit helps me to achieve the learning 
outcomes; the workload in this unit is appropriate to the achievement of the learning outcomes; the quality of 
teaching in this unit helps me to achieve the learning outcomes. Further, I am motivated to achieve the 
learning outcomes in this unit; I make best use of the learning experience in this unit; I think about how I can 
learn more effectively in this unit; and finally, overall, I am satisfied with this unit. In this study the 
researcher will examine and investigate if teamwork assessment and self/peer evaluation will assist students 
to improve and promote their learning skills (Curtin University n.d.). The learning skills are communication 
(writing, interpersonal interaction and cultural awareness, and presenting), critical and creative thinking 
(problem solving and decision-making), and Information Technology literacy and information literacy skills, 
as they have become crucial for information systems graduates. 


4. BT2 PARTICIPANTS 

In this study, the participants are 267 students who completed BT2 between 2006 and 2010. Most of these 
students are studying towards a double/single major in the Bachelor of Commerce degree (information 
systems and information technology, information technology and digital design, information systems and 
internet communications, information systems, and information technology). The BT2 focus is on operating 
systems and the networking basics, e.g. local area networks. A majority of BT2 students come directly from 
high school, and thus has little or no work experience. BT2 students are mainly from Australia and Asia 
(China, Hong Kong, Indonesia, Iran, Japan, Malaysia, Pakistan, Singapore, Sri Lanka, Taiwan, Thailand, and 
Vietnam); 75% are Asian, while 25% are Australian. It was noted that nationality and culture played an 
important role in the BT2 unit via the teamwork assessment, discussion, and debates in the lab session. The 
Asian students shared their knowledge in information technology and their computer experience by 
uploading the latest technology news and information on the blackboard, while the Australian students shared 
their work experience and understanding through discussions and debates. Feedback on BT2 confirmed that 
“learning to work with people with English being their second language always establishes a challenge but it 
has taught me some skills which I can use later when communicating with people from different countries”, 
and from the unit you “gain experience in how to work ‘with people of different cultures and countries’”. 


5. BT2 ASSESSMENTS 

The assessments for BT2 are divided into three: teamwork assessment and self/peer evaluation (30%), 
mid-semester test (30%), and the final exam (40%). These assessments focus on critical and creative 
thinking, problem solving and decision-making, and teamwork. The teamwork assessment consists of a set of 
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evaluation exercises (three to four questions) to familiarize students with the newest technology in the 
marketplace. These exercises are selected based on the most recent technology in the market to motivate and 
challenge students to explore and investigate the strengths and weaknesses of the new technologies in the 
marketplace, while the self/peer evaluation is designed to encourage students to think about and understand 
how well s/he and her/his partner performed the team activity, in order to develop communications and other 
skills. The mid-semester test and final exam consist of short-answer questions and case studies, in order to 
assess and examine students’ higher-level thinking skills and knowledge, on the basis of the unit learning 
objectives, the practical case study, and the lecturer’s formative and summative feedback during the 
semester. 


6. RESULTS 

Integrating teamwork assessment and self/peer evaluation in the BT2 unit was essential to promote and 
improve learning skills among BT2 students. This assessment assisted BT2 students to develop excellent 
relationships with their peers, sharing and exchanging ideas and knowledge in relation to the unit in general, 
and specifically to the assignment. This assessment has become the favorite for BT2 students, since it was 
well monitored and structured by the lecturer. From week one, the lecturer released the assessment 
information, namely assessment questions, presentation standard, guidelines for report writing, self/peer 
evaluation schedule, teamwork agreement and timeline, and finally the marking guide. In week three, the 
teamwork agreement, and timeline were submitted during the lab session to the lecturer. The lecturer and 
students must keep a signed copy of the agreement, and the lecturer monitored the agreement periodically. 
Integrating this assessment into BT2 prompts and enhances learning skills, develops good relationships 
between students, and encourages them to leant other skills besides the teamwork. It was noted from 
students’ formal and informal feedback that the teamwork assessment and self/peer evaluation was well liked 
among the students. Students confirmed that new knowledge and problem-solving, decision-making and 
technology skills were promoted and improved: “The teamwork assessment taught us how we should behave 
in a group discussion and how we should work to get the tasks finished quicker but with good quality 
results.” “Teamwork is the synergy that really helps in producing great stuff.” “Team work is about using 
other members' ideas along with your own to resolve a problem - as the saying goes, ‘two heads are better 
than one’”. Additionally, students were very pleased with the teamwork assessment, as they indicated: 
“Teamwork assessments are dynamic, and basically teach you how to handle people and their personalities.” 
“We made good friends for the future and have learned more about the topic.” Students learned and 
developed specific skills from this assessment beside the teamwork skills, and these skills will assist them in 
the current study and for the future workplace: i.e. problem solving, decision-making, cultural awareness, 
presentation, communication skills, and meeting a deadline. Students stated, “From this group activity we 
learn that it is very important to get along with someone. We need to support each other in our work. No 
matter from what background they come, everyone should be treated equally, since in real life experience, 
different people might have different points of view and it is very important to try to understand each other 
and support each other’s decision.” Furthermore, “although team work can be difficult, it helps with learning 
patience and being open to other peoples' ideas,” and “I have learned that showing respect and being patient 
is very useful in team working. Listen first and then give your opinion and try to show your appreciation for 
the effort that your team member has made.” 

Furthermore, other skills such as time management and meeting a deadline were promoted and developed 
among BT2 students by completion of the teamwork assessment. Students commented: “Doing team work 
and team assignment is a good way for me/us to learn how to prepare, how to manage my/our time, and it 
develops my/our ability to socialize and meet new people.” Students also confirmed that “working with the 
team, especially if s/he is a shy student, our member group will encourage me/us to ask questions during the 
meeting.” “It is important to determine pre-arranged deadline schedules. Communication is important; plan 
to finish work well before the due date to allow time to fix potential problems.” “We work together not only 
in the BT2 unit, but also in other units as a group, to complete other assessments.” BT2 students did not 
restrict their feedback to developing and promoting the skills area, but confirmed that completing this 
assessment will prepare them for the workplace and for their careers in the future. “The type of information 
generated from the assignment will help me throughout my career.” “The Team Working Assessment will 
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help me in participating in teamwork as we are going to face this teamwork in our real life.” Furthermore, to 
confirm the teamwork assessment’s success, students must listen to and respect their partner’s opinions, and 
students commented that “the Team Working Assessment teaches me how to work in a team and do work 
effectively to make a satisfactory outcome”. “Supporting and listening to other group members’ opinions is 
an important factor in group success.” From the informal feedback, students indicated that, without the 
lecturer’s commitment, encouragement, and formative feedback, the assessment could have failed. Students 
stated that the lecturer is “very energetic and full of current information”; “she tries hard to make the unit 
interesting; at the beginning of each lecture she always tells us what is happening in the news regarding IT.” 
In addition, “I like her for being able to resolve our learning to the practical in a real.” “She explains 
materials well and is friendly and approachable.” “She helps everyone who is in difficulties or having 
problems with the computer or the unit.” Finally, she “provides good real life situation news, articles and 
updates as well”, and. finally, our lecturer is “witty, clear, precise, and humorous”. Furthermore, it was noted 
that students were pleased with the BT2 learning experience and feedback, as well as expressing overall 
satisfaction (see Table 1). Table 1 indicates that the BT2 average in items 2, 5 and 11 in unit evaluation are 
higher than the university average. These outcomes provide the lecturer with more confidence to carry on 
with the current assessments, including the teamwork assessment and self/peer evaluation, and to impart 
formative and summative feedback to the students aligned to promote and improve their communication 
skills. 


Table 1 . eVALUate results - Unit Evaluations for BT2 from SI 2006 - S2 2010 (93% response rate) 


Year/Semester 

Students 

enrolled 

Response 

rate 

Item 2 

Learning experience 


Item 5 
Feedback 

Item 11 

Overall satisfaction 




Unit 

University 

Unit 

University 

Unit 

University 





Average 


Average 


Average 

2006/S 1 

54 

19% 

100 

81 

90 

70 

100 

78 

2007/S 1 

32 

47% 

100 

82 

93 

73 

93 

82 

2007/S2 

25 

56% 

93 

82 

86 

74 

86 

81 

2008/S 1 

29 

59% 

100 

83 

94 

76 

100 

82 

2008/S2 

26 

69% 

89 

82 

83 

77 

89 

83 

2009/S 1 

36 

47% 

100 

84 

100 

76 

100 

83 

2009/S2 

29 

62% 

89 

84 

83 

77 

88 

83 

2010/S2 

16 

56% 

100 

85 

100 

79 

100 

84 


Beside the quantitative results, students were keen to provide some qualitative responses in relation to the 
BT2 unit: “This is a pretty tough unit, but I think we can learn a lot of potential information from this unit for 
future use as an IS student.” Furthermore, “it is useful in application to real life”; “the unit is very helpful for 
the students who would like to continue their career in the IT area”; it is “very well organized, the material 
taught is well structured, organized, and well taught; lots of useful documents, lab notes etc. ... given to 
students; useful windows functions, facts, features taught”. Students also perceived the BT2 unit as “looking 
at the real life situation of the technology world and providing up-to-date information”; it has a “very logical 
and paced structure, the work is not over the top and you still are doing things.” The feedback from 
eVALUate (formal feedback) was not limited to the unit evaluation; students also presented their views on 
the teaching evaluation (see Table 2). 


Table 2. eVALUate results -Teaching Evaluations for BT2 from SI 2006 - S2 2010 (25% response rate) 


Question: The Lecturer: 

SI 06 

S 1 07 

S2 07 

SI 08 

S2 08 

SI 09 

S2 09 

S2 10 

Is well organized 

100 

100 

100 

100 

100 

100 

100 

100 

Communicates clearly 

100 

100 

78 

100 

77 

83 

90 

100 

Is approachable 

100 

100 

100 

100 

100 

100 

100 

100 

Provides useful feedback 

100 

100 

100 

91 

85 

100 

90 

100 

Appears knowledgeable in this subject area 

100 

100 

100 

100 

100 

100 

90 

100 

Is enthusiastic in teaching this unit 

100 

100 

100 

91 

93 

100 

100 

100 

Is an effective teacher 

100 

100 

100 

100 

85 

100 

90 

100 


Students offered the following comments about the lecturer: “The best aspect about the lecturer is that she 
is specific about her areas of assessment; this enables students to review those areas, and get back to her if 
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need be.” “She explains the lecture well and offers help when asked, and she is great when confronted to 
discuss a topic on an individual basis as she observes the level of the student and is able to adjust her 
explanations and examples to suit the student.” “She is very up to date with industry-relevant information 
regarding the unit, and is able to explain the concepts being taught in the unit clearly.” “The best lecturer I 
have ever seen. Easy to approach, provides very effective information for us, and she never delayed in 
response to our queries.” “[Our lecturer] is one of the best lecturers I have ever had! I am sure that all of the 
students will feel the same as me, she is the best lecturer at the university, no one is better than her.” The 
results from the mixed methods endorsed the study research question, as students confirmed, after completing 
the BT2 assessments, that they had learned new skills in the areas of communication ( writing, interpersonal 
interaction and cultural awareness, and presenting), critical and creative thinking (problem solving and 
decision making). Information Technology literacy, and information literacy. Finally, students corroborated 
that the new skills will be useful in university life, the future workplace and life in general. 


7. DISCUSSION AND LESSONS LEARNED 

This experience was interesting and an outstanding achievement from the lecturer's perspective, as the 
majority of the students indicated in their informal and formal feedback that completing the BT2 assessments 
allowed them to promote and develop their communication skills for university, future workplace, and life in 
general. The teamwork assessment and self/peer evaluation was designed on the basis of the literature review 
and the regular feedback from students. This assessment was a challenging exercise for the lecturer and the 
students simultaneously, as the lecturer provided formative feedback to her students in order to tackle any 
problems immediately and improve the presentation, structure, and layout of the assessment, while students 
were keen to present a weekly report regarding their assessment progress based on the team agreement. 
Additionally, from the lecturer's perspective, regular feedback prevents students from repeating the same 
mistakes and improves their learning behaviour and thinking skills, especially concerning report writing, 
research, and using Endnote software. On the other hand, summative feedback informs students about their 
level of performance when they reach the end of a topic or specific assessment task, or at the conclusion of 
the unit. As a result of using both formative and summative feedback, the number of requests for extensions 
dropped dramatically, and the extra time and effort that were put into the planning and organization of 
cooperative learning tasks were rewarded by an increase in students’ levels of satisfaction with the teaching 
and learning experiences. On the basis of the literature review and students’ feedback (Bates 2010 ; Gul & 
Boman 2006; Jaques & Salman 2007; Neus 2011; Nulty 2011; Wolcott et al. 2002 ; Yang, Newby & Bill 
2008), the author definitely recommends the integration of teamwork assessment and self/peer evaluation and 
time agreement in higher education. These assessments will encourage students to work as a team, and to 
improve their communication skills, including interpersonal interaction, negotiation, conflict resolution, 
listening, and patience with others, and their skills in the areas of leadership, brainstorming, research, writing, 
problem solving, and decision-making. Finally, on the basis of the students’ outstanding overall satisfaction, 
the author is now considered a teacher-leader in teamwork assessment in the school, and she now works with 
her colleagues to support and implement this assessment in the school curriculum strategy to match the 
university graduate attributes and to promote and improve students' learning skills. 


8. CONCLUSION 

This study has investigated the incorporation of teamwork assessment and self/peer evaluation in the higher 
education sector, specifically at the undergraduate degree level. The literature review confirmed the 
significance of this assessment in higher education, as students would improve their collaboration, 
cooperation, and communication skills, which have become essential for the lifelong learning process and 
future workplace. This paper discussed the strategies and techniques that were adopted for the teamwork 
assessment and self/peer evaluation in the BT2 unit. This assessment was a challenging exercise for both the 
students and the lecturer, as the lecturer always provided formative feedback to her students in order to tackle 
any problems immediately and improve the presentation of the assessment. The regular formative feedback 
prevents students from repeating the same mistakes and improves their critical thinking, self-confidence and 
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learning behaviour; also, students were keen to complete their tasks on time according to the team agreement. 
Furthermore, it was confirmed that students improved their communication skills, including interpersonal 
interaction, negotiation, conflict resolution, listening, and patience with others, and their skills in the areas of 
leadership, brainstorming, research, writing, problem solving, and decision making. Finally, it is 
recommended that this assessment should be implemented in higher education to meet the needs of students, 
university, and the workplace, as these skills are essential and critical in higher education in our time. In the 
future, the same exercise will be implemented for a postgraduate unit to examine the similarities and 
differences between students’ perception of teamwork assessment and self/peer evaluation. Furthermore, 
more research will be carried out to discuss the self/peer evaluation template, especially the strengths and 
weaknesses of the partner. 
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EVALUATION OF EPE VIDEOS IN DIFFERENT PHASES 
OF A LEARNING PROCESS 


Line Kolas, Robin Munkvold & Hugo Nordseth 
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ABSTRACT 

The goal of the paper is to present possible use of EPE videos in different phases of a learning and teaching process. The 
paper is based on an evaluation of EPE (easy production educational) videos. The evaluation framework used in this 
study, divides the teaching and learning process into four main phases: 1) The preparation phase, 2) The learning 
activities, 3) Follow up work and 4) Assessment. The evaluation of 132 EPE videos shows that the videos mainly are 
used in the preparation phase and/or as part of the learning activities. Videos used as part of formative or summative 
assessment are rare. The paper presents an illustration, which provides an overview of pedagogical use of educational 
videos, based on the four learning phases. Our findings show that all pedagogical opportunities concerning EPE videos 
still are not extensively used. 


KEYWORDS 

Educational video, evaluation framework 


1. INTRODUCTION 

The interest in short videos used for educational purposes, called EPE (easy production educational) videos is 
increasing. The definition of an EPE video is a pedagogical video clip (lasting up to 5 minutes) produced by 
the use of “low-effort” video equipment like mobile phones or small cameras. EPE videos should have a 
pedagogical purpose and be used in a pedagogical context. The goal of the paper is to present possible use of 
EPE videos in different phases of a learning and teaching process. The learning and teaching process is 
divided into four main phases: 1) The preparation phase, 2) The learning activities, 3) Follow up work and 4) 
Assessment. During the preparation phase, one motivates and raises interest in the learning objective(s), and 
the students prepare for the next phase. The next phase is traditionally connected to the classroom situation, 
where the teacher has prepared for a variety of learning activities, e.g. presentations, demonstrations, 
collaborative learning activities etc. During the 3rd phase “Follow up work”, the students continue the work, 
e.g. through homework. The 4th phase includes both formative assessment (assessment for learning) and 
summative assessment (assessment of learning). 

There are a variety of examples on how to use videos in the classroom, e.g. students watching 
films/videos, which presents a specific topic (Torgersen, 1998), active viewing of films (Newby et al, 2006), 
teacher recording classroom interactions e.g. the Marte Meo Method (Axberg et al, 2006), student- or 
teacher-generated videos e.g. digital storytelling (Lambert, 2009; Hpiland & Wplner, 2007; Olsen & Wplner, 
2003) and “low effort” use of mobile video technologies in the classroom (Kolas et al, 2011). Outside the 
classroom, videos can be used for e.g. preparation, using Khan Academy (http://www.khanacademy.org) or 
through methodology like “the flipped classroom” (Bennett et al., 201 1). 

The technology and software necessary to create, edit and present educational videos are widespread. 
Both teachers and students usually have access to a mobile phone camera, which can be used for recording 
educational videos. 90% of all Norwegian children aged 9-16 have their own mobile phone, and 97% of all 
Norwegian children aged 13-16 have their own mobile phone (Norwegian Media Authority, 2010). The 
Norwegian Monitor (Hatlevik et al, 2011) found that 75% of 13 years old children, 81% of 15 years old 
children and 86% of 18 years old Norwegians use their own computer at home. 99% of 13 years old children, 
98% of 15 years old children and everyone at the age of 18 have access to a computer at home. Editing 
software e.g. Windows Live movie maker, screencast software e.g. Screencast-o-matic.com or CamStudio 
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and publishing services like “YouTube”, are free and widely accessible. This is a good starting point for the 
use of EPE videos in a pedagogical setting, as the technology is already in their pockets or on their 
computers, and many of the students and teachers already are familiar to technology. 

The background of this paper is connected to two projects; “Naerproduksjon av video” (Developing EPE 
videos) 2011-2012, a national project carried out with the support of Norway Opening Universities focusing 
on how to produce and use EPE videos in teaching and learning settings; and “CoTech”, an international 
NordPlus project on collaboration technologies in education across borders. 

The structure of the paper is as follows. In section 2, we describe the methods, which the development of 
the evaluation framework is based upon. Then, a theoretical background of “educational videos” is provided. 
The next section briefly presents the evaluation framework before the findings of the study are presented in 
section 4 through figures and explanations. We then discuss the findings and present an overview of 
pedagogical use of videos in the fifth section, before the paper concludes in section 6. 


2. METHOD 

2.1 The Evaluation Framework 

The evaluation framework for EPE videos was developed based on literature review, evaluation of student 
tasks and through prototype testing by an expert panel. First, the literature review covered international 
literature on educational videos, with focus on categories/types of educational videos. Secondly, teacher 
students in three countries (Estonia, Denmark and Norway) as part of the CoTech project, first performed a 
task of planning, recording, producing and publishing an educational video, then through a follow-up task 
peer reviewed each other's videos. Teachers from teacher education institutions in the three countries have 
assessed both the student videos and the students’ peer reviews. The experiences of performing these tasks 
were important, as they visualized the needs of teachers who do not have experience of producing and using 
educational videos. Thirdly, an expert group was presented to a prototype of the evaluation framework that 
they tested on a variety of educational videos. The expert group consisted of ten persons with a technical 
and/or pedagogical background, who have experience with educational videos. The expert group's feedback 
was implemented into a new version of the evaluation framework. 

2.2 Evaluation of EPE Videos 

The evaluation was performed by 30 teachers/teacher students. They were told to find five educational 
videos, with the maximum length of 5 minutes, which they had used or would like to use in their classroom. 
During January 2012, the teachers and teacher students evaluated 4-5 educational videos each using the 
evaluation framework. 132 EPE videos were evaluated. The evaluation framework was available in digital 
form (through an online questionnaire), which was to be filled out, partly by free text, partly by the use of 
radio buttons and check boxes. To ensure a common understanding of all parts of the evaluation framework, 
the evaluation framework was introduced to all the participants using a “walk through” technique where the 
group evaluated the same educational video together. Any questions to the evaluation framework were 
discussed in the group to ensure a common understanding of the evaluation framework. 


3. THEORY 

3.1 Video Vitamins and the Flipped Classroom 

Educational videos can change the structure of teaching. Rismark et al (2007) describe how the teacher in a 
biology course prepared “video vitamins” (short video productions) with the goal that the students would 
show up better prepared in class compared to using traditional reading assignments. The videos were made 
available for the students through their learning management system and the students used mobile phones to 
watch the videos. The teacher prepared a video of 4-6 minutes together with the university multimedia centre. 
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using “greenscreen" technology. An alternative method of using educational videos for preparation is the 
“flipped classroom” methodology (Bennet et al, 2011), which is an “innovative classroom structure that 
moves the lecture outside the classroom via technology and moves homework and practice with concepts 
inside the classroom via learning activities” (Strayer, 2007). 

3.2 Formative and Summative Assessment 

Formative assessment means assessment for learning (Harlen, 2006) and is used to improve a student's 
learning process and learning outcome. Examples of formative assessment techniques are peer assessment, 
self-assessment, formative tests, visualization of demands, criteria, and progress. Summative assessment 
means assessment of learning (Harlen, 2006), and includes tests, exams, portfolios etc. 

3.3 Pre-, while-, post- Viewing Activities 

According to Lehiste (2012), videos are powerful classroom resources when employed appropriately. In 
order to exploit videos fully in the classroom, she suggests integrating pre-viewing, while-viewing and post- 
viewing activities. Examples of pre-viewing activities are to predict the content of the video based on the title 
or some keywords, to ask true -false statements, to generate questions about the topic (which the video is 
supposed to answer), and to brainstorm. With while-viewing activities, it is recommended that students watch 
carefully, not to miss important visual clues. The video clip can be played more than once if necessary. 
While-viewing activities include work sheets (e.g. with questions that can be answered during the video clip 
is showed), tables or schemes that can be filled out (Lehiste, 2012). Torgersen (1998) describes four types of 
worksheet activities: Detail-oriented tasks, problem-oriented tasks, describing tasks and visually oriented 
tasks. While -viewing activities can also be performed through alternative viewing techniques e.g. silent 
viewing (watching a video clip/scene with the sound off), sound only (listening for aural clues to the actions), 
freeze frame (to point out parts of the video, to ask questions etc), backwards viewing and jigsaw viewing 
(student partners will each know different, but incomplete versions of a story) (Lehiste, 2012). Examples of 
post-viewing activities are summarizing the video in own words, answering comprehension questions, 
comparing, creating a mind map, discussing, role-playing etc (Lehiste, 2012). 

3.4 Pedagogical Methods 

Heinich et al. (2002) categorize pedagogical methods into 10 main categories: drill and practice, presentation, 
tutorial, gaming, demonstration, discovery, problem solving, simulation, discussion and cooperative learning. 
In drill and practice, the learner is led through a series of exercises designed to practice/improve existing 
skills. The method assumes that the learner previously has been introduced to the theme. Drill exercises 
should include feedback to reinforce correct answers and remediate errors that learners might make (Heinich 
et al, 2002). In the presentation method a source tells, dramatizes, or otherwise presents information to 
learners. It is a one-way communication controlled by the source (teacher, book, video etc), with no 
immediate response from or interaction with learners (Heinich et al, 2002). Using the tutorial method a tutor 
(e.g. a person or computer software) presents the content, poses a question, requests a learner’s response, 
analyzes the response, supplies feedback and provides practice exercises until the learner demonstrates a 
predetermined level (Heinich et al, 2002). Gaming as a teaching method provides a playful environment 
where learners follow prescribed rules while working toward a goal. Gaming can be motivating, especially 
for tedious and repetitive content (Heinich et al, 2002). Alessi and Trollip (2001) define six game categories: 
adventure games, business games, board games, combat games, logical games, and word games. 

By demonstration, the learner views a real or lifelike example of the skill or procedure to be learned. The 
goal may be for the learner to imitate a physical performance or to adopt attitudes or values exemplified by 
someone who serves as a role model (Heinich et al, 2002). The discovery method uses an inductive approach 
to learning: it presents problems to be solved through trial and error. The problem solving method confronts 
the learner with a problem situated in the real world and the learner develops, explains and defends a solution 
on the problem (Heinich et al, 2002). This study considers discovery and problem solving as one pedagogical 
method. Learning through discovery and problem solving requires that learners search for information. 
Simulations involve learners confronting a scaled-down version of the real-life situation. It allows realistic 
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practice without the expense or risks otherwise involved (Heinich et al, 2002). Different types of simulations 
are physical, iterative, procedural, and situational simulations (Alessi & Trollip, 2001). 

As a pedagogical method, discussion involves the exchange of ideas and opinions among learners and 
instructor (Heinich et al, 2002). Cooperative learning involves learning together and learning from each 
other. Learners can collaborate, not only to discuss, but also by producing media (Heinich et al, 2002). 


4. FINDINGS 

The findings are based on the evaluation framework for EPE videos, which includes 24 questions. Theory 
and previous experiences are implemented into the evaluation framework. The evaluation framework focuses 
on five main topics: 1) Background information of the educational video. This includes name and duration of 
the video, where the video is published, subject and context of the video. 2) Added values of EPE videos. 
3) Different types of EPE videos using different technology, genres, technical effects, cognitive types, modes 
and learning phases. 4) The pedagogical methods the EPE videos are based upon. 5) The student versus 
teacher productions of EPE videos. This paper mainly presents results from topics 3 and 4, focusing on the 
pedagogical methods and use of the EPE videos in different teaching phases. 

4.1 Teaching/Learning Phases 

The teaching/leaming process is divided into four main phases: 1) The preparation phase, 2) The learning 
activities, 3) Follow up work and 4) Assessment. During the preparation phase, the teacher’s goal is to 
motivate and raise interest in the learning objective(s), and the students prepare for the next phase. The 
second phase is traditionally connected to the classroom activities, where the teacher has prepared for a 
variety of learning activities, e.g. presentations, demonstrations, collaborative learning activities etc. During 
the third phase “Follow up work”, the students continue the work, e.g. through homework. The fourth phase 
is assessment, and includes both formative and summative assessment. Figure 1 illustrates that most of the 
EPE videos were possible to use in the preparation phase and during learning activities, and a smaller number 
of videos can be used for assessment. 
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Figure 1. Number of EPE videos possible to use in the four 
different teaching phases. 



O phases 1 phase 2 phases3 phases4 phases 


Figure 2. Number of EPE videos that can be used in 
several teaching phases. 


The evaluation framework allowed multiple values to be filled in when categorizing the use of EPE 
videos into different teaching phases. Figure 2 illustrates how only 12 videos fit only one teaching phase. 118 
out of 132 videos were described to fit 2-4 different phases and two videos did not fit any of the teaching 
phases. This means that many of the videos can be used in e.g. both preparation phase and phase of learning 
activities. 
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4.1.1 Preparation Phase 

The evaluation framework asked about the intention of the EPE video if the video can be used in the 
preparation phase, and the results show that the main application areas in the preparation phase was to use the 
video for introduction and/or to motivate the students. Some videos were used to initiate reflections, while a 
small number of videos were used for reading support and/or as an artistic perspective to the learning goals. 



Figure 3. Application areas of videos used in the preparation phase. 

The evaluation framework allowed multiple values to be filled in when evaluating the application area of 
the videos use in the preparation phase. 76 out of 132 videos were described to fit more than one of the 
categories for the preparation phase, which means that a video can be used for e.g. both motivation and 
reading support. 

The “Other” category include explanations like “step by step instruction”, “preparation before using a 
tool”, “an example”, “visualization”, “introduction of a person” and “show the right technique”. 

4.1.2 Learning Activity Phase 

The EPE videos used in the learning activity phase were based on different pedagogical methods. The main 
pedagogical methods of the EPE videos were presentation and demonstration. Few videos are based on the 
following pedagogical methods: drill & practice, exploration, simulation, discussion, collaborative learning 
and game -based learning. 



Figure 4. Pedagogical methods used in the EPE videos in the learning activity phase. 


4.1.3 Follow up Work 

The ’’follow up work” phase includes e.g. the use of videos for homework. The findings show that half of the 
EPE videos are used for repetition (figure 5), while some EPE videos are used to summarize the classroom’s 
learning activities or to prepare for exams. 55 out of 132 videos were not possible to use as follow up work. 
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Figure 5. Application areas of EPE videos used for follow up work. 


The evaluation framework allowed multiple values to be filled in when evaluating the intention of the 
EPE videos in follow up work. 29 out of 132 videos were described to fit more than one of the descriptions, 
which means that a video can be used for e.g. both repetition and preparation for exam. 

4.1.4 Assessment Phase 

The assessment phase was divided into two categories: formative and summative assessment. 23 of the 
videos were found applicable for formative assessment (figure 6). Some of the videos could be used as video 
blogs for reflection (6 videos), self-evaluation situations (6 videos) or self-testing purposes (5 videos). Three 
of the videos also were adequate for video-based feedback from the teacher. Just two videos were adequate as 
pre- and post-video to visualize and measure the learning progress, and one video was possible to use for 
peer assessment purpose. Four of the videos had multiple possibilities for formative assessment. 



Figure 6. Use of EPE videos for formative assessment Figure 7. Use of EPE videos for summative assessment 

The evaluation shows that 33 of the videos were applicable for summative assessment (figure 7). Most 
applicable was use of video-based hand-in ( 1 1 videos) and use of video as a part of multiple-choice questions 
(8 videos). Seven of the evaluated videos could be used as for teacher generated observation and analysis and 
five could be used as a video-based journal for documentation. Six of the videos had multiple possibilities for 
summative assessment. 


5. DISCUSSION 

The results presented in this paper are based on an evaluation framework for EPE videos, which is developed 
as a tool for researchers to be able to characterize EPE videos for learning situations and a tool for 
teachers/teacher students who want to learn about multiple ways of varying the use of EPE videos in their 
teaching practice. This evaluation focuses on video resources chosen by the teacher in a process of 
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planning/teaching. It does not include student-generated videos, which also can be valuable learning 
resources. The results are based on the evaluation of 132 EPE videos. The number should be increased if the 
goal is to specify exact use of the educational videos. Six of the chosen EPE videos were evaluated twice; 
these videos are included because different teachers have different perceptions on how to use the videos in a 
teaching and learning situation. The findings presented should be considered as an attempt to start the work 
of defining the variety of EPE video use. The evaluation indicates that EPE videos tend to be used more in 
the preparation phase and the learning activities, than as follow up work and assessment. 

One of the reasons why many EPE videos are used for the preparation phase might be the influence of 
methodologies like the flipped classroom (Bennet et al, 2011) and video vitamins (Rismark et al, 2007) 
together with video web sites like Khan Academy and TED talks. Video is a well-known and useful tool for 
web-based tutorials, presentations and demonstrations. These are methods often used in a preparation phase. 
Another reason why many EPE videos are used for the preparation phase is that we want students to be 
prepared and to make the cognitive awareness start working on a certain topic. 

Based on the evaluation and our findings we present an illustration (figure 8), which provide an overview 
of opportunities when and how to use EPE videos. 



Preparation phase 
Motivation 
Initiating reflections 
Reading support 
Introduction /Lesson 
starter 
Artistic 
Example 


Learning acti\ities 

Drill and practice 

Presentation 

Demonstration 

Instruction with interactivity 

Game-based learning 

Exploration 

Simulation 

Discussion 

Collaborative learning 


Follow up work 
Summary- 
Repetition 
Preparation for the 
exam 

Video-based FAQs 


Assessment 


Formative assessment: 

Summative assessment: 

Video-based feedback from the teacher 

Video-based hand-in 

Pre- & post-video to visualize and 

Observation and analysis 

assess the student's development 

Video as part ofMCQ 

Peer assessment 
F ormative test (test yourself) 
Self evaluation 
Video-blogs for reflection 

V ideo -based j oumal 


Figure 8. The variety of EPE video use in different phases of the teaching and learning process. 

Flipped classroom methodology (Bennett et al, 2011) grows popular and represents the use of videos in 
the preparation phase. The flipped classroom methodology focuses mainly on the teacher’s 
presentation/introduction of new learning material. There are other pedagogical methods than presentation, 
which should be considered when planning to use EPE videos. The flipped classroom methodology mainly 
focuses on video use in the preparation phase, but EPE videos should also be considered in other phases of a 
learning process, e.g. as follow up work or assessment. 

Using EPE videos as part of the learning activities e.g. in the classroom, our findings indicates that EPE 
videos usually are based on presentation and demonstration as pedagogical methods. There are a number of 
other pedagogical methods that are not often used today. One may explain this by considering video as a 
useful tool when it comes to demonstration. When teaching e.g. a skill, the procedures can be described 
through text and still images in a book or through audio, as the teacher is explaining. Using a video one may 
include both a visual, oral and textual explanation of how to perform the skill. Motion pictures are usually 
more powerful than still images when the goal is to demonstrate a skill. 

Hopefully, the use of a variety of pedagogical methods will increase in EPE videos in the phase of 
learning activities. Teachers should become aware of the possibilities in the use of EPE videos, e.g. within 
mathematics, the use of simulation as pedagogical method might work as a tool for improving the cognitive 
understanding of a certain topic. Using EPE videos in order to visualize mathematical simulations -will 
hopefully make students better prepared for their classes and give them a better foundation to understand 
mathematical topics. Research within the field of mathematics shows that it is more efficient to work with the 
subjects based on an inquiry-based approach, making the students ask questions and wonder about why 
different mathematical formulas are defined the way they are, giving them a greater chance to understand the 
issue rather than just making use of a mathematical formula about which they do not understand the logics 
(Fuglestad, Goodchild & Jaworski, 2007, Jaworski, 2006). 

We also believe that EPE videos can be useful in follow up work. Examples of EPE videos as follow up 
work are to make EPE videos summarize students’ assessments, give them an overview of rights and wrongs 
and show good examples. This will give students the opportunity to use videos in different phases of their 
learning process, and provide visually-oriented students the opportunity to use their advantages for learning. 
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One of the reasons why only a few EPE videos evaluated are used for assessment might be because these 
videos are not necessarily widely published, as they mainly are useful only for a few persons. Feedback on 
hand-ins in many cases would be considered personal feedback, which is not relevant to publish on the web. 
A video blog for formative assessment, however, would probably be published open for anyone. 


6. CONCLUSIONS 

The paper has based on the results of an evaluation of 132 educational videos, presented possible use of EPE 
videos in different phases of a learning and teaching process. Our findings show that all opportunities 
concerning EPE videos still are not extensively used. We believe that teachers who in the future want to use 
EPE videos in their classes, should be aware of the opportunities of video use in different phases of a 
student’s learning process and the variety of different pedagogical methods an EPE video can be based upon. 
The paper presents an illustration, which describes an overview of pedagogical use of EPE videos. 
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ABSTRACT 

This study aimed to investigate the structural relationships among teaching presence, cognitive presence, usage, learning 
flow, satisfaction, and learning persistence in corporate e-learners. The research participants were 462 e-learners 
registered for e-lectures through an electronics company in South Korea. First, the sense of teaching presence, sense of 
cognitive presence, and usage affect flow. Second, the sense of teaching presence, sense of cognitive presence, usage, and 
flow affect satisfaction. Third, the sense of teaching presence, sense of cognitive presence, usage, flow, and satisfaction 
affect learning persistence. Fourth, learning flow intermediates the sense of teaching presence, sense of cognitive 
presence, usage, and satisfaction. Fifth, satisfaction intermediates the sense of teaching presence, cognitive presence, 
usage, flow, and persistence. We confirmed that learning flow significantly intermediated among sense of teaching 
presence, sense of cognitive presence, and satisfaction but not between usage and satisfaction. In addition, satisfaction 
intermediated among sense of teaching presence, sense of cognitive presence, usage, flow, and learning persistence. 
These findings demonstrate the importance of sense of teaching presence, sense of cognitive presence, and usage for e- 
learners. We expect that the results will contribute to the formation and improvement of fundamental learning strategies 
for successful e-learning. 


KEYWORDS 

Corporate e-learning, sense of teaching presence, sense of cognitive presence, usage, learning flow, satisfaction 


1. INTRODUCTION 

In an information society, where knowledge is a core resource, as the importance of human resource 
development increases incrementally, many enterprises emphasize employee education in order to improve 
their employee capabilities. Moreover, as people acknowledge the benefits of e-leaming — namely, that they 
can overcome time and space constraints through use of the Internet and other information and 
communication technologies — e-learning rapidly has become diffused and generalized. However, regardless 
of such quantitative growth, it can be difficult to actively engage online learners. Achieving flow in the e- 
learning process is challenging because e-learning differs from traditional education, which is conducted in a 
separate and private space. Thus, online learners may not be motivated to continue due to low learning 
satisfaction. As advocates have called for investigation of the e-learner’s experience in the e-learning process 
with the aim of improving its quality, researchers have sought to understand the sense of presence and its role 
in e-learning. 

Sense of presence is expressing not what exists in the physical environment, but rather what one 
experiences and perceives (Witmer & Singer, 1998). Sense of teaching presence refers to a learner’s 
perception regarding a general teaching phenomenon, along with aspects of designing and systemizing 
instruction (Arnold & Ducate, 2006). Sense of cognitive presence is defined as the degree of consistent and 
confirmed meaning in a learner’s reflection and discourse (Garrison & Anderson, 2007). The current study 
did not adopt the notion of sense of social presence, but did adopt senses of teaching presence and cognitive 
presence as effective variables for learning outcomes. 

First, this study attempted to clarify the cause and effect relationships among sense of teaching presence, 
sense of cognitive presence, flow, satisfaction, and persistence, within an integrated view. Despite the 
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importance of sense of teaching presence, sense of cognitive presence, and usefulness for e-learners, the 
research investigating their effects on learning outcomes from an integrated view remains insufficient. 
Previous studies related to successful e-learning investigated simple correlation or effects between two 
variables rather than integrating the variables related to learning persistence. For example, studies have 
examined the relationship between teaching presence and satisfaction (Wu & Hiltz, 2004) as well as the 
effect of cognitive presence on satisfaction and presence (Kang, 2005), of teaching presence on persistence 
(Shin, 2003), of usage on satisfaction and persistence (Roca, Chiu, and Martinez, 2006), of usage on 
persistence (No, Lee & Chung, 2008), of flow on satisfaction (Kim, 2005), of cognitive presence on 
satisfaction and achievement (Kang, 2005), and of flow on satisfaction and achievement (Kang, 2006). Since 
learning outcomes in cyber-environments, such as satisfaction and learning persistence, are complex 
phenomena affected by assorted variables(Willging & Johnson, 2004), observers should consider the relevant 
variables’ causal relationships with an integrative structural view. 

Second, previous studies on the relationships among flow, satisfaction, and persistence, using SEM, have 
not addressed important variables in predicting successful e-learning. Joo, Kim, and Park (2009) did not 
include the sense of teaching presence in e-learning but did suggest the necessity of investigating it. Kim 
(2006) investigated the relationships among usefulness, ease of use, and behavioral and attitudinal flow but 
did not include the outcome variables of flow. Chiu, Hsu, Sun, Lin, and Sun (2005) investigated the 
relationships among usefulness, satisfaction, and persistence with the intermediation effect of satisfaction but 
did not include the teaching and cognitive sense of presence prior to looking at usefulness and outcome 
variables. Moreover, the results of previous studies are somewhat contradictory. Thomas’ (2000) 
investigation of the effects of students’ social networks on learning persistence in a university setting found 
that the relationships among social networks, scholastic integration, social integration, grades, flow in 
objectives, flow in educational organization, and learning persistence were not significant. In research 
investigating the relationships among satisfaction, confidence, flow, and intention to repurchase on 615 e- 
learners in various institutional settings, Lee (2006) showed that flow does not affect the persistence to 
purchase. 

The purpose of this study is to explain the relationships among the sense of teaching presence, usage, and 
learning outcomes, which facilitate learner flow in corporate e-leaming, by integrating all the variables in a 
single structural model. Based on a review of the previous literature, we established research hypotheses and 
hypothetical research models of corporate e-learning. First, the sense of teaching presence, sense of cognitive 
presence, and usage affect flow. Second, the sense of teaching presence, sense of cognitive presence, usage, 
and flow affect satisfaction. Third, the sense of teaching presence, sense of cognitive presence, usage, flow, 
and satisfaction affect learning persistence. Fourth, learning flow intermediates the sense of teaching 
presence, sense of cognitive presence, usage, and satisfaction. Fifth, satisfaction intermediates the sense of 
teaching presence, cognitive presence, usage, flow, and persistence. Based on a review of the previous 
literature, we established research hypotheses and hypothetical research models of corporate e-learning. 
These are shown in Figure 1 . 

First, the sense of teaching presence, sense of cognitive presence, and usage affect flow. Second, the 
sense of teaching presence, sense of cognitive presence, usage, and flow affect satisfaction. Third, the sense 
of teaching presence, sense of cognitive presence, usage, flow, and satisfaction affect learning persistence. 
Fourth, learning flow intermediates the sense of teaching presence, sense of cognitive presence, usage, and 
satisfaction. Fifth, satisfaction intermediates the sense of teaching presence, cognitive presence, usage, flow, 
and persistence. 



Figure 1 . Hypothetical Research Model of Corporate e-Learning 
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2. METHOD 

2.1 Subjects 

We chose to investigate a single enterprise for the consistency of its registration system and learning 
management system. We issued a survey to 462 participants to measure sense of teaching presence, cognitive 
presence, usage, learning flow, satisfaction, and learning persistence. Male participants numbered 375 
(81.2%), and females, 87 (18.8%). The participants’ ages ranged from 24 to 54. Their job statuses were as 
follows: 29.5% staff, 21.4% deputy section chiefs; 17.3% section chiefs, 15.1% deputy department heads, 
7.4% department heads, and 9.2% other. 

2.2 Research Instrument 

To measure the sense of presence, we used the validated instrument designed specifically for measuring 
sense of presence by Garrison, Cleveland-Innes, and Fung (2004), extracting only the sense of teaching 
presence and sense of cognitive presence. The sense of cognitive presence is measured by eight items. For 
both variables, the inter-item consistency had a Cronbach's a of .94. We removed the second item in one 
duplicated items out of the 13 items measuring sense of teaching presence. Thus, the final measurement used 
12 items for sense of teaching presence and 8 for sense of cognitive presence. The construct reliability of 
sense of teaching presence was .99, and the average extracted variance was .99. The construct reliability for 
sense of cognitive presence (the reliability of the dormant variable) was .95, and the average extracted 
variance was .91. We measured usage by extracting items from Davis’(1989) Technology Acceptance Model 
instrument. Usage consisted usefulness and ease of use. Usefulness and ease of use consisted of four items 
each. Inter-item consistency had a Cronbach's a of .87 calculated for usefulness and .86 for ease of use. The 
construct reliabilities of usefulness and ease of use were both .99, and the average extracted variance was .97. 
We used 9 items from the Flow state scale instrument validated by Martin and Jackson (2008) to measure 
learning flow. The inter-item consistency for learning flow items had a Cronbach's a of .83 for participants in 
high school gym class and .84 for those in high school music class. The construct reliability was .99, and the 
average extracted variance was .99. To measure this variable, we revised Shin's (2003) measurement 
instrument for corporate settings. The instrument consists of 8 items rated on a 5-point Likert scale. The 
inter-item consistency had a Cronbach's a of .96, the construct reliability in the current study was .99, and 
the average extracted variance was .99. The instrument measuring learning persistence consisted of 4 items. 
The construct reliability in the current study was .97, and average extracted variance was .99. 

2.3 Research Procedure 

To collect data, we conducted an online survey for e-leamers enrolled in courses at Enterprise A. We selected 
Enterprise A because it uses the same registration and management systems, learning service, and evaluation 
and grade generation systems for its various sub-branches. We administered the survey the week prior to the 
e-leamers completing the four-week course. The main instructional methods were lectures given by the 
instructor, delivered by video. To investigate the causal relationship between sense of teaching presence and 
sense of cognitive presence in corporate e-learning, we established the hypothetical research model shown in 
Figure 2 (see Section 1.5) and the statistical model in Figure 1. As seen in the statistical model, we 
established each mathematical dormant variable using index variables from the research model. 
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Figure 2. Statistical Model of Corporate e-Learning 

In establishing the model, we used an item parcel method to avoid overweighting on the measurement 
model, since there are single -factor measurement variables in exploratory factor analysis results among sense 
of teaching presence, sense of cognitive presence, usage, learning flow, satisfaction, and learning persistence. 
To determine the prediction method for the statistical model, we examined multivariate normal distributions 
of 8 variables of the SEM, using AMOS 6.0. As a result, we were able to satisfy the conditions of skewnesses 
and kurtoses for single variables. We predicted the model fitness and parameters using a Maximum 
Likelihood Estimation (MLE) procedure, given that the multivariate normal distribution assumption was 
satisfied. We evaluated the model fitness through CMIN, TLI, CFI, and RMSEA. 


3. RESULTS 

The variable means ranged from 3.55 to 4.22, standard deviations from .66 to .73, skewnesses from .12 to .67, 
and kurtoses from .10 to .30. This satisfied the basic assumptions of SEM, as the skewnesses of the 
measurement variables were less than 3, and their kurtoses were less than 10. Therefore, the variables 
satisfied the basic assumptions of a multivariate normal distribution for SEM examination. 

Since the initial fitness of the structural model was TLI = .977, CFI = .986, and RMSEA = .068 (.054- 
.082), we confirmed that the fitness index of the initial structural model indicated it was a good model. 
Accordingly, we examined the direct effects among sense of teaching presence, sense of cognitive presence, 
usage, flow, satisfaction, and learning persistence. The results were as follows: sense of teaching presence on 
flow was P = .413 (t = 8.200, p < .05); sense of cognitive presence on flow, P = .41 1 (t = 6.174, p < .05), and 
usage on flow, ft = .122 (1 = 2.027, p < .05). Second, the effects of sense of teaching presence on satisfaction, 
P = .109 (t = 2.091, p < .05); cognitive presence on satisfaction, p = .272 ( t = 4.054, p < .05); and usage on 
satisfaction, p =.144 (t = 2.550, p < .05). Third, the effects of sense of teaching presence on learning 
persistence, p =.136 (t = 12.687, p < .05); and cognitive presence on learning persistence, p =.797 (t = 
12.687, p < .05). The effect of usage on learning persistence was not statistically significant. The initial 
structural model of this study reveals the significant fact that removing the paths between sense of cognitive 
presence and learning persistence, between usage and learning persistence, and between learning flow and 
learning persistence did not affect the model fitness. Accordingly, we established the simple model seen in 
Figure 3, in which we removed the paths mentioned. 
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Figure 3. Revised Research Model of Corporate e-Learning 

Because the initial structural model and revised model have hierarchical relationships, we conducted a 
chi-square test to determine whether there was a statistically significant difference between the two. The 
result showed no difference between the models in their goodness of fit (CMIN D = 5.389, p - . 145). 
Accordingly, although there was no difference in goodness of fit between the models, we selected the revised 
model and estimated the goodness of fit and parameters, since the revised model was simpler. <Table 1> 
shows the results of the revised structural model’s goodness of fit. By confirming the goodness of fit index, 
we confirmed that the model fitness was good (CMIN D = 5.389, p = .145). 


Table 1. Examination Results of Fitness of Corrected Model (n = 462) 



CMIN 

df 

TLI 

CFI 

RMSEA (90% Confidence Interval) 

Corrected Structural Model 

126.815 

42 

.978 

.986 

.066 (.053-.080) 

Initial Structural Model 

121.426 

39 

.977 

.986 

.068 (.054-.082) 


As seen in <Table 2>, the model's overall fitness index appeared similar to the initial structural model, 
but the absolute fitness index value x 2 (chi-square) was 5.389 higher. The fitness indices were statistically 
significant; all the fitness indices of the revised structural model satisfied the fitness criteria. In addition, the 
sense of teaching presence and satisfaction had significant effects on learning persistence. <Table 2> 
summarizes the direct model effects. 

Table 2. Direct Model Effects 


Direct effect 


Non-standardization 

Error 

of C.R. 

P 

Standardization 

Flow 

f— Teaching presence 

.40 

.05 

8.22 

* 

.41 


Cognitive presence 

.42 

.07 

6.18 

* 

.41 


Usage 

.14 

.07 

1.97 

* 

.12 

Satisfaction 

s— Teaching presence 

.11 

.05 

2.01 

* 

.11 


Cognitive presence 

.27 

.07 

3.84 

* 

.26 


Usage 

.19 

.07 

2.73 

* 

.16 


Flow 

.46 

.06 

7.14 

* 

.44 

Persistence 

f— Teaching presence 

.14 

.04 

3.65 

* 

.16 


Satisfaction 

.75 

.04 

17.92 

* 

.83 

> < .05 


111 





ISBN: 978-989-8533-12-8 © 2012 IADIS 


These results of this study show as followings: first, the sense of teaching presence, sense of cognitive 
presence, and usage had significant effects on learning flow. Second, sense of teaching presence, sense of 
cognitive presence, and usage had significant effects on satisfaction. Third, satisfaction also had significant 
effects on learning persistence. Also, learning flow had significant effects on satisfaction, and satisfaction 
had significant effects on learning persistence. Fourth, we found that sense of teaching presence, sense of 
cognitive presence, and usage had significant effects on satisfaction by intermediating learning flow. Fifth, 
sense of teaching presence, sense of cognitive presence, and usage were found to have significant effects on 
learning persistence by intermediating satisfaction. This shows the possibility that learning flow has 
significant effects on learning persistence by intermediating satisfaction. Accordingly, the Sobel test was 
applied to examine the significance of the indirect effects. <Table 3> displays the indirect effect analysis of 
the variables affecting learning outcomes. 


Table 3. Direct and Indirect Effects of the Modified Model (n = 462) 



Relevant variables 

Total effect 

Direct effect 

Indirect effect 


Teaching presence 

.40 

.40* 

- 

Flow 

<— Cognitive presence 

.42 

.42* 

- 


Usage 

.14 

.14* 

- 


Teaching presence 

.30 

.11* 

.18* 

Satisfaction 

Cognitive presence 

.46 

.27* 

.19* 

Usage 

.25 

.19* 

.06 


Flow 

.46 

.46* 

- 


Teaching presence 

.22 

.14* 

.08* 


Cognitive presence 

.20 

- 

.20* 

Persistence 

<— Usage 

.14 

- 

.14* 


Flow 

.35 

- 

.35* 


Satisfaction 

.75 

.75* 

- 


*p < .05 


4. DISCUSSION 

Based on the research results, we find the following: First, we confirmed that sense of teaching presence, 
sense of cognitive presence, and usage have significant effects on learning flow in corporate e-leaming. The 
significant effects between teaching and cognitive presence on learning flow are consistent with previous 
research findings (Barfield, Zeltzer, Sheridan, & Slater, 1995; Wang & Kang, 2005). The significant effect of 
usage on learning flow is also consistent with previous research results (Kim & Oh, 2005). 

Second, we confirmed that sense of teaching presence, sense of cognitive presence, usage, and flow have 
significant effects on satisfaction in corporate e-learning. The significant effects of teaching presence 
(Garrison & Cleveland-Innes, 2005; Wu & Hiltz, 2004; Shin, 2003) cognitive presence (Joo, Kim, & Park, 
2009; Kim, 2008), usage (Chiu, Flsu, Sun, Lin & Sun, 2005; Roca, Chiu, & Martinez, 2006), and flow on 
satisfaction (Shin, 2006) are consistent with previous research results. 

Third, we confirmed the significant effects of sense of teaching presence and satisfaction on learning 
persistence in corporate e-learning. The significant effect of teaching presence on learning persistence is 
consistent with previous research findings (Shin, 2003). The significant effect of satisfaction on learning 
persistence is consistent with previous studies that reported if learners are satisfied with overall aspects, such 
as the instructor, teaching method, process, and learning environments, they are likely to continue their 
learning after completing the course. On the other hand, in this study, sense of cognitive presence, flow, and 
usage did not directly affect persistence. This finding is not consistent with previous studies, which reported 
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opposite results (Joo, Kim, & Park, 2009). This suggests the possibility that the sense of cognitive presence 
does not directly affect learning persistence. Further, findings on the effect of usage on persistence is not 
consistent with previous studies (No, Lee, Chung, 2008). Although previous studies (Chiu, Hsu, Sun, Lin, & 
Sun, 2005; Roca, Chiu, & Martinez, 2006) did not observe direct effects of usage on learning persistence, 
they reported that usage significantly affects learning persistence by intermediating satisfaction. Then, the 
findings regarding the effect of learning flow on learning persistence are also not consistent with previous 
studies, which reported opposite results, namely, that flow does affect persistence in sports environments 
(Kim & Lee, 2008). These results likely differ due to the environmental distinction: Flow in a sports 
environment means that participants are deeply engaged in attitudes or behavioral aspects of the sport rather 
than in cognitive processes. 

Fourth, satisfaction intermediated among sense of teaching presence, sense of cognitive presence, usage, 
learning flow, and learning persistence. These results mean that when learners' perceived sense of teaching 
and cognitive presence improve, they can experience flow. Accordingly, learners’ overall satisfaction 
improves, and learning persistence in taking relevant courses increases even after the students complete the 
current course. This finding is consistent with the previous research results (Chiu, Hsu, Sun, Lin, & Sun, 
2005). 

The suggestions and contributions from the current research results are as follows: First, e-learning 
designers, therefore, should increase learners’ sense of teaching presence by providing them with 
opportunities to ask questions about class content as part of the learning process, in order to confirm what the 
learners know and to correct any misunderstandings. Educators should also encourage each learner’s 
persistent participation by managing his or her learning process through e-mail, short message services, or a 
webpage. Second, to increase the sense of cognitive presence, e-leaming designers should structure learning 
content appropriately, allow learners themselves to generate new knowledge, and develop systems to help 
learners manage their learning resources and time. Third, e-learning designers, therefore, must improve usage 
by providing practical cases closely related to the students’ work, so learners can feel that the system is very 
useful and easy for them to use. Fourth, e-learning designers, therefore, should improve usage by devising 
strategies to raise flow in the learning process, improve learning outcomes, avoid distractions for learners, 
and consider the learners’ convenience so that they are not hindered in their studies. Fifth, thus, e-learning 
designers, should increase learner satisfaction by emphasizing strategies that use the e-leaming design and 
management elements, learners’ interest in the learning process, and a systematic educational process to 
improve the quality of educational content and to avoid inconveniences for learners. 

The limitations of the study and our suggestions for further research studies are as follows. First, the 
results of this study have a limited generalizability: we used 462 participants, all employees of Enterprise A 
in South Korea, who received an employment insurance refund of the course cost. Future studies should 
investigate whether different e-learning settings or cyber-universities produce the same results. Second, since 
we expect that the perceived degree of a sense of cognitive presence differs for individual learners, further 
research is needed to investigate various motivational variables, rather than simply reflect learner 
characteristics. Third, we administered the study to reflect corporate e-learning characteristics, after 
removing sense of social presence and focusing on sense of teaching presence and sense of cognitive 
presence. However, future studies should consider the sense of social presence that learners experience in e- 
learning environments, because successful learning occurs through the integration of the three: sense of 
teaching presence, sense of cognitive presence, and sense of social presence. Fourth, we removed learning 
achievement from the learning outcome variable because learners received the employment insurance refund 
from Enterprise A only if they gave more than 70% correct answers. Therefore, we decided that it would be 
somewhat difficult to explain learning achievement as a learning outcome. However, since achievement is an 
important variable for measuring learning outcomes, further studies should include achievement as a learning 
outcome variable. The ultimate purpose of corporate educational training is not a better grade but to improve 
employees’ performance through their application of the knowledge and skills obtained in e-learning. We 
suggest future research include studies on learning transfer or Return On Investment. 
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ABSTRACT 

This paper presents language learners as social networkers and describes and discusses the types of users that can be 
identified by analysing the content created by them using a situated mobile language learning app, LingoBee, based on 
the idea of crowd sourcing. Borrowing ideas from other studies conducted on social network users, we can identify that 
language learners use LingoBee as a social network and they behave as social networkers by creating content, acting as 
Conversationalists, Critics and displaying other behaviours shown by social networkers. In addition to this, from our user 
studies, it can be seen that the language learners are stimulated by the contributions of other users as well as welcoming 
competition among users. LingoBee users as social networkers were analysed and discussed based on Luckin's idea of the 
Zone of Proximal Assistance and the Zone of Available Assistance (Luckin, 2009). 
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1. INTRODUCTION 

Social Networks and crowd sourcing have recently received a lot of attention in the area of Technology 
Enhanced Learning, introducing new concepts along with the ideas of learning as a social activity and 
communities of learners (Haythornwaite, 2011). Learners who already engage with social networks such as 
Facebook expect similar capabilities in their learning environments, sometimes even expecting the learning 
processes to be connected in some way to their social networks. This opens up a whole new dimension to the 
evaluation of learning and learning processes, drawing in ideas from Social Network Analysis and other 
disciplines to the area of Technology Enhanced Learning. This brings new insights and new means of 
understanding learning and the behaviour of individual learners as well as communities, or more 
appropriately, networks or crowds of learners (Altshuler, Fire, Aharony, Elovici, & Pentland. 2012). 

In this paper, we present some user studies that we have conducted in several different countries, where 
the learners were provided with a mobile language learning app, LingoBee, to support their learning process. 
LingoBee is based on ideas that have been explored in an earlier project, in a mobile app called Cloudbank 
(Pemberton, Winter, & Fallahkair, 2010), which has a wiki-like functionality to crowd source language 
content from anonymous users. User studies conducted on Cloudbank showed that users were also social 
networkers. They showed that experience of applications such as Facebook played an important role in 
learners’ expectation; e.g. some users expressed their wish to have an identity and desired functionalities to 
support multiple contributions, comments and ratings to "bubble up" good content (Pemberton & Winter, 
2011). This experience highlights the role that social media plays in our lives and learning processes and how 
users' expectations are being affected by it. Thus, LingoBee included functionality to support social 
networking among users such as a user profile and identity as well as wiki-like contents and rating of 
content. 

The aim for this paper is to identify whether LingoBee users act as a true social network or some kind of 
hybrid and to identify and describe the types of LingoBee users and the implications this has on how best to 
support teaching and learning, to positively impact on learners’ engagement and use of LingoBee. 
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The rest of this paper is organized as follows: Section 2 provides an outline of the functionality of 
LingoBee; Section 3 outlines the general study design for the three user studies; Section 4 describes the data 
collection methods and the types of social networkers that are relevant for LingoBee; Section 5 presents 
evidence of users as social networkers; Section 6 presents and discusses implications for learning; Section 7 
summarises the paper. 


2. LINGOBEE MOBILE APP 
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Figure 1. LingoBee functionality, a) LingoBee repository; b) definition of an entry; c) editor to enter a definition d) user 
profile; e) browsing: search, filter, sort; c) a flagged definition 

LingoBee is a mobile app to support situated mobile language learning and to help the learners in linguistic 
and cultural diversity. Based on the ideas of situated learning (Lave & Wenger, 1991) and contextualized 
learning, e.g. (Luckin, 2010), it is designed to capture language elements that learners come across in their 
everyday lives, whenever and wherever. Ideas of crowd sourcing and social networking are used to collect. 
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share and annotate the contributions of all learners in a shared online repository as shown in Figure la). 
Users are able to add entries, which may be words or phrases, to the LingoBee repository, which can be 
accessed by other users of LingoBee, see Figure lb), which shows a definition containing a picture, and 
Figure lc) where the user can enter new definitions. Learners are also able to add new definitions to existing 
entries and rate existing definitions; e.g. in Figure lb) the entry "lenticchie" has one definition and users have 
rated it as five stars. Each entry can contain multimedia elements such as a picture and/or audio content as 
well as web links. In addition, a text-to-speech functionality is available for the correct pronunciation of the 
entries. Language articles are co-constructed through the use of LingoBee, where students add items and 
build meaning together through dialogue created via multiple entries. 

To support social networking, LingoBee provides the capability for the users to define their profiles such 
as the username and contact details as shown in Figure Id). Users are able to view or browse through the 
contents of the LingoBee repository and the functionalities to support browsing are shown in Figure le). 
Other ideas from social networks have been included such as peer rating as shown in Figure lb) and flagging 
an entry as a form of feedback as shown in Figure If). 

LingoBee was developed as part of an EU LLP project SIMOLA, with partners from six different 
countries, (SIMOLA, 2012b). Thus, LingoBee is supported in 6 different European languages and lapanese. 
It is a mobile app developed for the Android platform. 


3. USER STUDIES 

User studies of LingoBee have been conducted in several locations (Italy, UK, Norway, Lithuania, Hungary 
and lapan) since July 2011. The user studies presented in this paper were carried out in three different 
European educational establishments: Bellerbys College, Oxford, part of Study Group UK (Study Group), 
The University of Molise (Unimol), Italy, the Department of Social Science and Linguistic Centre and 
Department of Languages and Communication at the Norwegian University of Science and Technology 
(NTNU). The users were all language learners, enrolled on university or pre-university courses. 

The study design involved introducing LingoBee to international students learning English, Italian and 
Norwegian, and providing them with a free smartphone for use in their daily lives. LingoBee was formally 
introduced to the students in a classroom setting, although it did not always form any part of the classroom 
activities. In general, the students were provided smartphones with the LingoBee app pre-installed. LingoBee 
was introduced in different ways: by presenting the basic functionalities for adding content to LingoBee, 
through demonstrating the functionality through a video, through the help guides uploaded onto a VLE and 
through activities designed to encourage students to explore the functions of the app. 

Considering the users as motivated, independent learners, which is one of the findings of the studies 
conducted by (Pemberton & Winter, 2011), one of the aims when introducing LingoBee was to show the 
students the basic functionality without influencing them with our views of the app and its usage. Another of 
the aims of evaluation was to see how the users perceived the systems and used the functionality. 


4. METHOD 

The results presented in this paper are based on the content in the LingoBee repository and the pre- and post- 
intervention questionnaires and interviews conducted with the participants in the studies at Study Group, 
Unimol and NTNU. In addition to these, data logs created by the LingoBee system were used to determine 
some of the activities of the users that were not visible from the LingoBee mobile app interface. Google 
Analytics were also used to detect activities of users. However, for the results that are discussed in this paper, 
we have not analysed the data from Google Analytics in detail. Our focus so far has been on the content 
created by the users in the repository. 

In this paper, the main analysis has been to identify LingoBee users as social networkers. Based on the 
motivations for the design of LingoBee and the work conducted in analyzing the types of Social Networking 
users such as The Social Technographic Ladder by the Forrester Research Inc. ("The Social Technographics 
Ladder," 2011-2012), we have identified the following types of users as relevant for LingoBee and 
potentially other social networking based language learning apps: 
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• Creators: users that create entries in the LingoBee repository by adding new words, phrases, additional 
definitions and multimedia content. 

• Conversationalists: users that add entries onto other users' definitions. 

• Critics: users that provide peer reviews by using the rating and flagging functionality on content in the 
LingoBee repository. 

• Collectors: users that add other users' definitions from the LingoBee repository to their favourites list. In 
LingoBee, an individual user's wordlist or favourites are on their mobile device. These are entries 
created by them and the entries that they have downloaded from the repository to their favourites list. 

• Spectators: users that viewed content in the LingoBee repository. 

• Inactives: users that were none of the above or users who were active users at the beginning and then 
stopped. 


5. LINGOBEE USERS AS SOCIAL NETWORKERS 


Based on the results of the studies at Study Group, Unimol and NTNU, we can categorise LingoBee users as 
social networkers, using the categories described in Section 4. We will illustrate the different types of users, 
using examples sourced from the LingoBee repository, the post-intervention questionnaires and interviews. 
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Informal use. 
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d) e) 0 

Figure 2. Definitions of “mint”: a conversation between a learner and the teacher 

The obvious category of user is the Creator as evidenced by the LingoBee repository, which is accessible 
from (SIMOLA, 2012a) and examples of which are presented in Figure la), b) and f). It is interesting to note 
that Creators were of two main types: i) true creators that created entries autonomously and adopted 
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LingoBee as a natural part of their language learning, adding an entry when they saw something new and ii) 
those that required prompting to add new content. The analysis of our results concluded that the initial 
studies at all three locations experienced the second type of Creators. Thus, prompts were used to stimulate 
the creation of more content, such as tasks set by the teacher, e.g. scavenger hunts, or a Facebook group to 
support each other. 

In the same way, there seem to be two types of Conversationalists: i) true conversationalists that interact 
and exchange content taking into consideration what others previously said, just like in any other real life 
conversation, and ii) those who simply add entries in a row, without being influenced by others’ content. The 
example in Figure 2 shows a "Conversation" where users have created collective definitions in a wiki-style. 
Such users are Conversationalists, where one user's definition of an entry is complemented by another user’s 
definition. In this particular example, a conversation between the teacher and a student is shown. This is also 
a prime example of how a conversation between the learner and a native speaker could take place through 
LingoBee, where LingoBee users include native speakers as well as learners. There are several other 
examples where one user's entries have additional definitions. For example, in the Unimol user group the 
following entries were made: 

3 different entries for SEDIA (chair): 

04/04/2012,B,sedia, 

22/04/2012,F,sedia, 

03/04/2012, Si,Se, sedia 

4 different entries for POLLO (chicken) 

19/1 1/201 1,E, polio al curry, 

29/1 1/201 l.PJPollo, 

01/12/201 l.PJPollo, 

1 9/04/20 1 2, S .Polio, 

These are not true conversations; they are multiple attempts at one single definition by different users in 
the Unimol trials. It is interesting to see such examples. Here, the users are in fact Creators of new entries. 
However, it appears that they are unaware of existing entries, thus creating a new entry rather than adding to 
existing definitions, i.e. a conversation. Had the entry "Polio" existed before the user wished to add the entry 
"polio al curry", would it have been a conversation? These are the types of questions that we focused on 
answering through our interviews. With regards to this point, we can observe that this is due to a feature of 
LingoBee, it detects automatically identical entries and links them together, but it does not inform the user 
that there is already an entry in the repository. If it did warn the user, they may be less inclined to make a 
separate entry. 

Examples of Critics can be seen in Figures 1, 2 and 3; users have rated entries in the LingoBee repository, 
e.g., in Figure lb) and Figure 2a), the entries have a rating of five stars. Similarly, users have flagged content; 
e.g. in Figure If), the entry is flagged as it is spelt incorrectly. Flagging could indicate several things such as 
incorrect spellings, inappropriate or abusive content. Both learners and teachers acted as Critics; e.g. the 
entry in Figure If) was flagged by the teacher to draw the learners' attention to the incorrect spelling. 

Collectors are not obvious from the LingoBee interface. However, we asked in the post-intervention 
questionnaire ‘Is it helpful to see the words and phrases added to LingoBee by other users’ rated on a Likert 
scale of 1-7, where 7 indicates the highest and 1 indicates the lowest level of agreement. The combined 
results, across all three studies showed that 68% of all respondents agreed and that 42% strongly agreed 
(rated 7) with this statement. This, combined with the logs created by the system, identified entries from the 
repository that have been added to other users' favourites lists. Examples of Collectors are shown in 
Figure 3; both the entries have only one definition, but you can see from c) from the data collected by 
LingoBee, that they have been added to favourites 6 and 3 times respectively. It is not possible to detect 
Spectators from the LingoBee interface. However, based on the data from the Google Analytics (which we 
are analysing at the time of writing), the post-intervention questionnaires and interviews, it is clear that all 
three studies had users who were Spectators that browsed the LingoBee repository. The Google Analytics 
data will also help to find out how many times an entry has been viewed but not added to favourites. The 
post-intervention questionnaire (which we are currently analysing) had questions such as the number of hours 
a learner used LingoBee during the day and the level of LingoBee usage in different locations such as home, 
city, etc., on a Likert scale of 1-7. The data from the questionnaires shows that some students reported a 
higher number of hours of using LingoBee per day compared to the level of activity shown on the logs in 
terms of them as Creators, Conversationalists, Critics and Collectors. We believe this is an indication that 
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some users are Spectators. In addition to this, some of the learners that have been interviewed reported 
face-to-face discussions related to LingoBee content with other LingoBee users in their language class. 
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Figure 3. a) Definitions of Train spotter b) Definition of Creme de la creme c) Extracts from the logs showing number of 

times these entries have been added to a users favorites list 

Some users were Inactives as they joined the study and accepted a smartphone with LingoBee, but either 
never used LingoBee or stopped using it during the course of the user study, as they stopped attending the 
language classes for various reasons. 

We are currently conducting an in-depth analysis of the data and therefore we have not supported our 
observations reported in this section with statistics. In addition to the types of users discussed so far, the 
response to the post-intervention question "What additional functionalities would you like in LingoBee to 
support language learning?" provided a few interesting responses such as "widget that shows best 
definition/information ranked, so people can compete (against) each other, to be ranked on the table....", "It 
could be more interactive like a network. Maybe using questions about the name of some objects and doing a 
competition between the users with a game" and "motivation" . These responses from the learners illuminate 
two key points, the first that students like the idea of using LingoBee in a similar way to which they use other 
social media, e.g. number of likes on Facebook or Instagram or playing competitive games via Facebook and 
they could possibly be motivated by this. If students tend to think it could be more “ like a network”, maybe 
LingoBee is not a real social network, at least not in the sense they would like. It currently has basic social 
network functionalities, but it could allow more interaction and be more integrated with other networks - e.g. 
allowing automatic integration with Facebook. Furthermore, within the growing family of social media, 
LingoBee appears to resemble interest-focused networks like Instagram or Pinterest, where you do not 
necessarily search for friends but for people sharing the same interest. 

The second point is that a competitive spirit is indicated as a means of motivation and recognition among 
peers. This identifies language learners as a new type of social networker: Competitors and Motivators. 
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6. IMPLICATIONS FOR LEARNING 


In this section, we analyse the types of LingoBee users from the perspectives of the work produced in the 
area of Technology Enhanced Learning, in particular, learning contexts. An interesting approach is 
developed by Luckin, whose ideas are based on the Zone of Proximal Development proposed by (Vygotsy, 
1978). Luckin added two additional concepts by defining a Zone of Proximal Adjustment and a Zone of 
Available Assistance, (Luckin, 2009). The Zone of Available Assistance describes a variety of assistance that 
can be made available to a learner, whilst the Zone of Proximal Adjustment represents a selection from the 
Zone of Available Assistance for a given learner and the educational situation. 
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Figure 4. The Zone of Collaboration around LingoBee users adapted from (Luckin 2010) 

LingoBee users as social networkers or as a community of learners can be considered in the context of the 
ideas proposed by Luckin; as shown in Ligure 4. The tools available to users within the Zone of Available 
Assistance are: other LingoBee users; native speakers; the teacher as well as a myriad of tools and 
technologies, such as online dictionaries providing potential assistance to the learners. The key to engaging 
learners in the user studies, was ensuring LingoBee and other LingoBee users moved from the Zone of 
Available Assistance into the Zone of Proximal Adjustment. The first trial group at Study Group struggled to 
have any tools in their Zone of Proximal Adjustment, they had never used mobile phones in formal learning, 
they weren’t able to use the VLE and they were too new to the college to understand the importance of being 
able to use these. Their main tools or available assistance was their teacher, email and text messages and the 
LingoBee app (only when supported to use it). In the second Study Group user study, through class 
discussions, it became apparent that the entries by one particular user was liked by the other participants of 
the study and stimulated them to learn the language and use LingoBee, thus this user became a More Able 
Partner (Luckin, 2010) or at least another tool in their Zone of Proximal Adjustment. Similarly, some of the 
entries by users in the NTNU user study were rated by the teacher, encouraging and promoting those users 
and similar entries in the LingoBee repository. Similarly, the example in Ligure 2 shows how the teacher can 
provide support to a learner. What is important is that the network of learners and through ideas of social 
networking, LingoBee users could provide implicit support to one another in their learning process. Thus, the 
LingoBee users can play a significant role in both the Zone of Proximal Adjustment and the Zone of 
Proximal Development as a More Able Partner. 

The Luckin diagram works as a powerful tool for teachers and researchers. It helps to assess the 
preliminary situation in order to prepare the setting to introduce the app. It also highlights the constraints that 
could have a negative impact on the learning process. In any case, the experiences of our studies show that 
current users’ expectations of social networking cannot be ignored as they are part of their mind-set and 
orientate them in creating, selecting and sharing learning objects. 

To enable LingoBee users to provide relevant support to one another, there is a need "to have a big group 
(using LingoBee )" , as stated by one of the users in their post-intervention questionnaire. Our earlier user 
studies indicated that the learners required assistance in starting to use LingoBee, thus requiring scaffolding 
initially by the teacher in various ways. Lor example, in addition to the teacher being a Conversationalist to 
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prompt the users. Study Group introduced various activities to support learning, e.g. a scavenger hunt, 
walking tour, show and tell activities (often around food), access to a VLE and tasks to engage with the target 
culture such as watching specific television programmes. Similarly, the second study conducted at Unimol 
introduced a Facebook support group to motivate and encourage the users. Our studies suggest, there is a 
need for scaffolding when using LingoBee and when engaging users with the other tools in their Zone of 
Proximal Assistance (Wood, Bruner, & Ross, 1976), (Vygotsy, 1978) and (Luckin, 2010). 


7. SUMMARY 

This paper presents LingoBee users as social networkers and describes and discusses the types of users that 
can be identified by analysing the content created in the LingoBee repository and the data that is available. 
Borrowing ideas from other studies conducted on social network users, we can identify that LingoBee 
language learners use LingoBee as a social network. As social networkers they are: Creators of content; 
Conversationalists, Critics of other users' entries; Collectors that download entries created by other users; as 
Spectators that browse the content as well as Inactives. In addition to this, from the post-intervention 
questionnaires and interviews, it can be seen that the language learners are stimulated by the contributions of 
other users and welcome competition. LingoBee users as social networkers were analysed and discussed 
based on Luckin’s idea of the Zone of Proximal Assistance and the Zone of Available Assistance. 

The next stage of our work is to analyse the data that has been gathered in more detail and to provide 
statistical evidence. In addition, we will analyse the data for a better understanding of language learners as 
social networkers and the role of crowdsourcing and social networking in language learning to provide better 
support to language learners. 
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ABSTRACT 

Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide aiming to 
create, share and reuse digital educational resources among educational communities. As a result, organizing, offering 
and accessing these resources over the web have been key issues for both the research and the educational community. 
Traditionally, a popular way for characterizing digital educational resources is by using a formal and centrally agreed 
classification system, such as the IEEE Learning Object Metadata (LOM). On the other hand, with the emerging Web 2.0 
applications, the issue of characterizing digital educational resources tends to move from the expert-based description 
based on formal classification systems to a less formal user-based tagging referred to as social tagging. As a result, a 
number of studies have been reported in the field of Technology-enhanced Learning (TeL) aiming to evaluate whether 
social tagging can improve the discovery of new digital educational resources and the retrieval of known ones stored in 
web-based repositories. Additionally, recent studies in the field of social tagging systems has provided initial evidence 
that users’ tagging motivation has a direct influence on the properties of the resulted tags and folksonomies. Thus, in this 
paper we aim to propose a methodology for investigating this issue and examine whether different end-users’ tagging 
motivations (that is teachers) could enlarge the metadata descriptions of digital educational resources. The results of our 
study provided us evidence that there is a direct influence of users' tagging motivation on the enlargement of metadata 
descriptions of digital educational resources, as well as to the resulted folksonomy when it is compared with formal 
structured vocabularies used by metadata experts or content providers for characterizing digital educational resources. 


KEYWORDS 

Educational resources, social tagging, folksonomy, user tagging motivation, evaluation methodology, evaluation results 


1. INTRODUCTION 

Over the past years, several Open Educational Resources (OERs) initiatives have been emerged worldwide 
towards the provision of open access to digital educational resources, in the form of Learning Objects (LOs) 
(McGreal, 2008). UNESCO (2002) has defined Open Educational Resources (OERs) as the “ technology- 
enabled , open provision of educational resources for consultation, use and adaptation by a community of 
users for non-commercial purposes ”, The main objective of OERs initiatives is to support the process of 
organizing, classifying and storing digital educational resources and their associated metadata in web-based 
repositories which are referred to as Learning Object Repositories (LORs) (McGreal, 2004). Within this 
context, a popular way for describing digital educational resources is by using a formal and centrally agreed 
classification system, such as the IEEE Learning Object Metadata (LOM) (IEEE LTSC, 2005). 

On the other hand, the emerging Web 2.0 applications have led to an enormous increase of the digital 
resources available on the web today. As a result, both the discovery of new resources and the retrieval of 
known ones on the web, become an increasingly complex problem (Heymann, Koutrika and Molina, 2008; 
Yande, Jatowt, Nakamura and Tanaka, 2007). Therefore, the issue of characterizing digital resources tends to 
move from the expert-based description based on formal classification systems to a less formal user-based 
tagging (that is, adding keywords to digital resources) (Derntl, Hampel, Motschnig-Pitrik and Pitner, 2011; 
Bi, Shang and Kao, 2009). Adding keywords, also known as tags, to any type of digital resource by users 
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(rather than resources’ authors) is referred to as Social Tagging (Bonino, 2009). The term of social tagging 
has emerged for those applications that encourage groups of individuals to openly share their private 
descriptions (or tags) of digital resources with other users, either by using a collection of tags created by the 
individuals for their personal use (referred to as folksonomy) (Anderson, 2007). 

In the field of Technology Enhanced Learning (TeL), social tagging has also been proposed as a mean for 
describing digital educational resources (Cho, Yeh, Cheng and Chang, 2011; Bateman, Brooks, McCalla and 
Brusilovsky, 2007). As a result, the generation of metadata is now done by individual users, who might 
primarily see private benefits, like an easy way to search and retrieve from LORs already used and known 
resources using meaningful terms (Dahl and Vossen, 2008). Thus, a number of studies have been reported 
aiming to evaluate the added value of socially tagging digital educational resources stored in LORs and 
compare it with the traditional approach of expert-based formal description based on centrally agreed 
classification systems, such as IEEE LOM (Trant, 2009b; Vuorikari and Ayre, 2009). Additionally, recent 
studies in the field of social tagging systems suggests that users’ tagging motivation has a direct influence on 
the properties of resulting tags and folksonomies (Korner, et al., 2010; Korner, 2009) but there are not 
existing studies for investigating this issue in the field of TeL. 

To this end, in this paper we aim to investigate this issue and we propose a methodology that aims to 
evaluate whether users’ tagging behaviour can influence (a) the enlargement of metadata descriptions of 
digital educational resources and (b) the resulted folksonomy when it is compared with formal structured 
vocabularies used by metadata experts or content providers for characterizing digital educational resources. 
The application of the proposed evaluation methodology in an existing LOR, namely the 
OpenScienceResources Repository (http://www.osrportal.eu/) provided us evidence that ‘verbose’ taggers 
can produce tags that are significantly different from formal metadata generated by metadata experts or 
content providers. 

The paper is structured as follows. Following this introduction. Section 2 provides an overview of 
existing studies from the literature, which have investigated the potential benefits of socially tagging digital 
educational resources. In Section 3, we present our proposed methodology for identifying different types of 
users’ tagging motivation and evaluating their possible influence to the metadata descriptions of digital 
educational resources, as well as to the resulted folksonomy when it is compared to formal structured 
vocabularies used for metadata authoring by metadata experts or content providers. Section 4, presents the 
tagging approaches adopted by the existing LOR (namely OpenScienceResources) where we applied our 
proposed methodology, the dataset of the OpenScienceResources repository that was available at the time of 
our study, as well as the evaluation results and discussion of our findings. Finally, we present our conclusions 
and ideas for future work. 


2. LITERATURE REVIEW 

Within the TeL literature there are existing works that have examined the added value of socially tagging 
digital educational resources. An initial study towards addressing this issue has been conducted in the 
framework of the EU-funded project “MELT: a Metadata Ecology for Learning and Teaching” 
(http://info.melt-project.eu/) (Zens et al., 2009; Vuorikari and Ayre, 2009). More specifically, MELT project 
developed a LOR including around 153.000 digital educational resources. These resources were 
characterized by the resources’ authors with educational metadata following an application profile of the 
IEEE LOM standard, as well as with social tags added by the end-users of the repository (that is students and 
teachers). During the evaluation study of the project, the MELT Repository included 10131 social tags, 
which had been added to 2008 digital educational resources. This means that on average 4 social tags were 
added per digital educational resource. The main issues that were investigated during the MELT project 
evaluation study were the following: 

■ II: usefulness of social tags for searching digital educational resources. 

■ 12 : usefulness of social tags as metadata descriptors of digital educational resources. 

The evaluation results in respect to the aforementioned issues showed that: 

■ II: 23% of searches in MELT Repository were performed based on social tags and 40% of these 
searches were found very useful. However, for accurate searches within MELT Repository, social 
tags were considered not useful. 
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■ 12 : 25% of social tags contained additional to authors’ information, 26% unnecessary information 
and 49% no new information. Moreover, social tags were found to be more useful than formal 
vocabulary terms and most of the end-users wanted to change the original metadata description to 
adopt some of the social tags as indexing terms. 

Another important study about social tagging has been conducted in the framework of the US-funded 
project “STEVE: The Museum Social Tagging Project” (http://www.steve.museum/) (Trant, 2009a). More 
specifically, the project developed a repository including around 97.000 digital resources of cultural heritage. 
These resources were characterized with metadata by professional museum experts as well as with tags added 
by the end-users of the repository (that is students and teachers). During the evaluation study of the project, 
the STEVE Repository was including 36981 tags which had been added to 1792 digital cultural heritage 
resources. This means that on average 20 social tags were added per cultural heritage resource. The main 
issues that were investigated during the STEVE project evaluation study were the following: 

■ II: social tags correlation with museum metadata. 

■ 12 : usefulness of social tags as museum metadata descriptors of the digital cultural heritage 
resources. 

The evaluation results in respect to the aforementioned issues showed that: 

■ II: 86% of social tags didn’t match with museum metadata added by professional museum experts. 

■ 12 : 88% of social tags were considered useful by the professional museum experts 

As we can notice from the aforementioned studies, there is a strong interest for investigating the added 
value of social tagging on enlarging the metadata descriptions of digital educational resources, as well as the 
formal vocabularies used in expert-based metadata. However, both issues have been investigated without 
considering the possible implications of users’ tagging motivation to the enlargement of resulting tags and 
folksonomies. Next, we aim to address this issue and we present our proposed evaluation methodology. 


3. PROPOSED EVALUATION METHODOLOGY 

In this section, we present our proposed methodology for identifying different types of users’ tagging 
motivation and evaluating their possible influence to the metadata descriptions of digital educational 
resources, as well as to the resulted folksonomy when it is compared to formal structured vocabularies used 
for metadata authoring by metadata experts or content providers. More specifically, our methodology 
includes the following steps: 

■ Step 1 - Identify different underlying behaviours for users’ tagging: This step includes the 
discrimination of the users based on their tagging motivation. For this purpose, we adopt two types 
of user motivations proposed by Komer (2009): 

o Categorizers, who are motivated by categorization and use tags to construct and maintain a 
navigational aid to the resources they annotate. For this purpose, categorizers aim to 
establish a stable and bounded vocabulary based on their personal preferences and 
motivation. 

o Describers, who are motivated by description aim to describe the resources they annotate 
accurately and precisely. As a result, their tag vocabulary typically contains an open set of 
tags. 

In order to discriminate between categorizers and describers we adopt a set of measures proposed by 
Korner et al. (2010): 

o Tag/Resource Ratio : relates the vocabulary size of a user to the total number of digital 
educational Resources tagged by this user. Describers, who use a variety of different tags 
for their resources, score higher values for this measure than categorizers, who use fewer 
tags. 

o Orphaned Tag Ratio: characterizes the degree to which users produce orphaned tags (that is 
tags assigned to few resources only, and therefore are used infrequently). The orphaned tag 
ratio captures the percentage of tags in a user's vocabulary that represent such orphaned 
tags. Categorizers vocabulary scores values closer to 0 because orphaned tags would 
introduce noise to their personal vocabulary, whereas describers vocabulary scores values 
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closer to 1 due to the fact that describers tag resources in a more verbose and descriptive 
way. 

o Overlap Factor: measures the phenomenon of an overlap produced by the assignment of 
more than one tag per resource on average. Categorizers are interested in keeping this 
overlap relatively low. On the other hand, describers produce a high overlap factor since 
they do not use tags for navigation but instead aim to best support later retrieval. 
o Tag/Title Intersection Ratio: measures how likely users choose tags from the words of an 
educational resource title. Categorizers use tags taken from the title and they score values 
closer to 1, whereas describers rarely use tags from the title and they score values closer to 
0 . 

■ Step 2 - Calculate similarity between social tags and educational metadata: During this step we 
calculate the similarity between social tags (offered by end-users, that is teachers) and educational 
metadata (offered by metadata experts or content providers). The similarity is calculated for social 
tags added by describers, as well as for social tags added by categorizers based on the users’ 
discrimination performed in step 1 . At the end of this step, we expect to identify digital educational 
resources enlarged with social tags offered by describers and/or categorizers that are different by the 
formal metadata descriptions offered by metadata experts or content providers. 

■ Step 3 - Compare folksonomy with formal vocabularies of educational metadata: During this 
step, we compare the resulted folksonomy produced by the social tags with formal structured 
vocabularies of educational metadata. The comparison is performed with the folksonomy produced 
by describers, as well as with the folksonomy produced by categorizers following the users’ 
discrimination performed in step 1. At the end of this step, we would be able to identify new tags 
offered by describers and/or categorizers that can enlarge the formal structured vocabularies of 
educational metadata. 


4. APPLYING PROPOSED METHODOLOGY TO AN EXISTING 
LEARNING OBJECT REPOSITORY 

In this section, we apply our proposed evaluation methodology to an existing LOR, namely 
OpenScienceResources Repository (http://www.osrportal.eu/). The OpenScienceResources Repository was 
developed in the framework of an EU-funded project, referred to as “OpenScienceResources: Towards the 
development of a Shared Digital Repository for Formal and Informal Science Education” 
(http://www.openscienceresources.eu/). It provides access to openly licensed (through Creative Commons) 
science education resources, which can be used by science teachers connecting formal science education in 
schools with informal science education activities taken place in European Science Centres and Museums 
(Sampson, Zervas and Sotiriou, 2011). 

The OpenScienceResources Repository currently includes around 3000 science education resources 
characterized by content providers (European Science Centres and Museums) with educational metadata 
following an application profile of the IEEE LOM standard (Sampson, Zervas and Sotiriou, 2011). More 
specifically, the application profile of the OpenScienceResources Repository includes formal structured 
vocabularies at the Classification Element (Nr. 9) of the IEEE LOM standard for describing (a) the subject 
domain of science education resources that is related with the science curriculum (the vocabulary includes 
400 terms) and (b) the educational objectives that science education resources intend to target (the 
vocabulary includes terms selected by Bloom’s Taxonomy of Educational Objectives and its subsequent 
revisions and extensions (Anderson and Krathwohl, 2001)). 

The science education resources of the OpenScienceResources Repository are also characterized with 
social tags added by the end-users of the repository (that is teachers). The social tags includes the following 
categories: (a) free tags, that are mainly used by the end-users, so as to describe the topic and/or the subject 
domain of a science education resource related with the science curriculum and (b) educational objectives 
tags, that are used by the end-users, so as to express the educational objective(s) that a science education 
resource can target. The educational objectives tags are selected from a predefined vocabulary following 
revised Bloom’s Taxonomy of Educational Objectives and it is the same with the formal vocabulary, used by 
content providers for characterizing science education resources with IEEE LOM compliant educational 
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metadata. Table 1 presents the two tagging approaches of the OpenScienceResources Repository and how the 
metadata elements added by content providers can be compared with the social tags added by end-users. 

Table 1. OpenScienceResources Repository Tagging Approaches 


Educational Metadata (according to IEEE LOM) 


Metadata Element 
1 .5 General. Key word 

9. 2. 2. 2 Classification.Taxon 
Path.Taxon.Entry (when purpose 
metadata element has the value 
“discipline”) 

9. 2. 2. 2 Classification.Taxon 
Path.Taxon.Entry (when purpose 
metadata element has the value 
“educational objective”) 


Value Space 

Free text describing the topic of 
the science education resource 
Structured vocabulary describing 
the subject domain of the science 
education resource related with 
the science curriculum 

Structured vocabulary describing 
the educational objectives that a 
science education resource intends 
to target 


Social Tags 


Tags Categories 
Free Tags 


Educational 
Objectives Tags 


Value Space 
Free text describing 
the topic and/or the 
subject domain of a 
science education 
resource related with 
the science 
curriculum 

Structured vocabulary 
describing the 
educational objectives 
that a science 
education resource 
intends to target 


Next, we present the snapshot of the OpenScienceResources Repository dataset at the time of our study 
(March 2012) that was used for applying our proposed evaluation methodology described in Section 3. 

Table 2. OpenScienceResources Dataset (March 2012) 


Variables 

Value 

Taggers 

434 

Tagged Science Education Resources 

1877 

Social Tags 

14707 

Free Tags 

13117 

Educational Objectives Tags 

1590 


4.1 Describers and Categorizers 

In order to cluster the taggers of the OpenScienceResources Repository we applied the set of measures 
described in our proposed methodology in Section 3. Table 3 presents the number of categorizers and 
describers resulted by applying these measures. 

Table 3. Describers and Categorizers 


Type of Taggers 

Value 

% per Total Taggers 

Categorizers 

226 

52,07 % 

Describers 

208 

47,92 % 

Total 

434 

100,00 % 


Additionally, Table 4 presents quantitative data about the number of tags used and the number of science 
education resources tagged by each type of tagger. 


Table 4. Quantitative Data per Type of Tagger 


Type of 

Number 

Tags 

Average Tags 

Science Education 

Average Tagged 

Average Tags 

Taggers 

of 

Taggers 

Contributed 

per Tagger 

Resources Tagged 

Science Education 
Resources per 
Tagger 

per Science 
Education 
Resources 

Categorizers 226 

1960 

8,67 

1852 

8.19 

1,05 

Describers 

208 

12647 

60,8 

630 

3,02 

20,07 


As it was expected we can notice from Table 4 that describers contributed the vast majority of social tags 
of the OpenScienceResources dataset but they have tagged less science education resources than the 
categorizers. This means that describers mainly use a small set of digital educational resources, which they 
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want to accurately and precisely describe for later searching and retrieval. On the other hand categorizers 
contributed a small amount of social tags to a large set of digital resources aiming to support future browsing 
to as many educational resources of the repository. 

4.2 Educational Metadata vs. Social Tags 

The next step of our proposed evaluation methodology includes similarity calculation between educational 
metadata and social tags. More specifically, we calculated (a) the similarity of the metadata values added by 
content providers to the element Nr. 1 .6 Keyword of the IEEE LOM standard with the free tags category of 
the social tags added by describers and categorizers and (b) the similarity of the metadata values added by 
content providers to the element Nr. 9.2.2.2 Classification. TaxonPath. Taxon. Entry (when purpose metadata 
element has the value “educational objective”) of the IEEE LOM standard with the educational objectives 
tags category of the social tags added by describers and categorizers. Table 5 presents average similarity 
results for the different type of taggers, as well as the number of science education resources that achieved 
low similarity score in average (<0,5). This similarity threshold was selected by considering relevant 
thresholds used in other social tagging evaluation studies from the literature (Vuorikari and Ayre, 2009). 


Table 5. Educational Metadata vs. Social Tags 


Type of Tagger: Keyword vs. 

Classification.Taxon 

Science Education 

% per total tagged 

Free Tags 

Path.Taxon.Entry vs. 

Resources with low 

science education 


Educational Objectives 

similarity score (<0,5) 

resources 


Tags 



Categorizers 0,79 

0,71 

184 out of 1852 

9,93% 

Describers 0,76 

0,45 

307 out of 630 

48,73% 


As we can notice from Table 5, describers added social tags (mainly to the educational objectives tags 
category) that were significantly different from the educational metadata added by the content providers. 
Therefore, they have significantly contributed to the enlargement of the metadata descriptions of 307 science 
education resources. This was the 48,37% of the science education resources that they have tagged. On the 
other hand, categorizers added tags that were quite similar with the educational metadata added by the 
content providers. Therefore, this distinction between categorizers and describers is very important because it 
facilitates capturing enlarged metadata descriptions of digital educational resources. 

4.3 Folksonomy vs. Formal Structured Vocabulary 

During the final step of our proposed evaluation methodology, we compared the resulted folksonomy of free 
tags category of social tags added by describers and categorizers with the structured formal vocabulary used 
by content providers to characterize science education resources with terms related with science curriculum. 
In order to achieve that, we calculated the similarity of the describers folksonomy and categorizers 
folksonomy with the formal structured vocabulary and we kept those tags that they achieved similarity score 
zero. The number of these tags was 202 (from describers) and 94 (from categorizers). Finally, we excluded 
the semantic noise from these tags, that is synonyms and subjective tags and we concluded to 46 (describers) 
and 10 (categorizers) possible new terms that could enlarge the structured formal vocabulary used at the 
element Nr. 9.2.2. 2 Classification. TaxonPath. Taxon. Entry (when purpose metadata element has the value 
“discipline”) of the IEEE LOM standard. Finally, we investigated whether there were new terms contributed 
by both describers and categorizers. Table 6 presents the distribution of the new terms contributed. 


Table 6. Distribution of new terms per type of tagger 


Type of Taggers 

New terms Contributed 

% per Total Terms 

Only by Describers 

46 

82,14 % 

Only by Categorizers 

10 

17,85 % 

By Categorizers & Describers 

0 

0,00 % 


As we can notice from Table 6, describers outperformed categorizers by contributing 36 more new terms. 
Moreover, there were not common terms contributed by both describers and categorizers. These results 
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provided us evidence that describers have a stronger influence on the enlargement of formal structured 
vocabularies, whereas categorizers’ contribution was limited. These findings could be relevant in LORs for 
indexing digital educational resources with describers’ tags towards facilitating search and retrieval of digital 
educational resources. 


5. CONCLUSIONS AND FUTURE WORK 

In this paper, we investigated the influence of different tagging styles, namely describers and categorizers on 
enlarging the metadata descriptions of digital educational resources. The results of our study performed with 
the dataset of an existing LOR, namely OpenScienceResources Repository showed that describers produce 
tags that are significantly different from formal metadata, whereas categorizers mainly follow the formal 
metadata generated by metadata experts or content providers. As a result, considering tagging motivation 
during folksonomies analysis could facilitate capturing enlarged metadata descriptions of digital educational 
resources. Future work includes deeper analysis to the results of our study, so as to identify the effect of 
describers’ and categorizers’ social tags to the enlargement of metadata descriptions for digital educational 
resources with different granularity levels and different formats. 
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ABSTRACT 

Wiki technology provides new opportunities to foster collaborative learning in various educational settings. To 
empirically examine the impact of wikis on learning, this article explores students’ collaborative writing activities 
performed on MediaWiki. The activities were analyzed using a taxonomy with ten categories (clarify content, add 
content, delete content, add link, delete link, fix link, grammar, spelling, style/typography, and formatting). The work 
also analyses students’ comments posted on the discussion page of the wiki. The results show important differences in the 
types of contributions across the categories investigated. The results also reveal that the level of collaboration and 
discussion was relatively low compared with other activities performed on the wiki. Finally, the article suggests a number 
of factors influencing wiki-based collaborative writing in teacher education. 


KEYWORDS 

Collaboration, collaborative learning, collaborative writing, MediaWiki, wiki. 


1. INTRODUCTION 

With the emergence of wiki technology, opportunities for fostering group interaction and participation are 
greatly enhanced. Combined with the collaborative learning theory and the socio-constructivist view of the 
learning process, wikis are increasingly becoming potentially powerful tools to foster collaboration among 
participants (Caple & Bogle, 2011; Grant, 2009; Li & Zhu, 2011; Tetard, Packalen & Patokorpi, 2009; 
Thomas, King, & Minocha, 2009). Wikis offer possibilities for teachers and learners to collaborate on joint 
assignments and collective writing tasks. They enable active participation of contributors by editing and 
revising each other’s writings, and hence contributing to the sharing of knowledge and collective 
development of wikis. However, although teachers and students are encouraged to exploit the possibilities 
offered by wikis in their practices, they may not be confident or experienced in supporting collaborative 
writing, or adapt the technology to the characteristics of the learners and target audience. This paper reports 
on a study that focuses on wiki technology as a tool for collaborative writing in teacher education. The 
overall aim is to assess the activities carried out on the wikis by means of the history function that tracks all 
students’ contributions. Particular attention was devoted to students editing one another’s contributions. The 
activities are analyzed quantitatively using a taxonomy proposed by Pfeil, Zaphiris, and Ang (2006). The 
work also analyses students’ comments posted on the discussion page of the wiki. Finally, influencing factors 
are discussed to exploit the opportunities offered by wiki technology to foster collaborative writing in teacher 
education. 
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2. BACKGOUND 

2.1 Wiki Technology 

This work used one of the most popular wiki platforms - MediaWiki - to perform collaborative writing 
activities (Kasemvilas, & Olfrnan, 2009). MediaWiki uses a simplified HTML language and provides an 
extensive functionality for user authentication (Su & Beaumont, 2010). Another important functionality of 
MediaWiki is the history function that keeps track of students’ edits by name, date, and colour coding (Lund 
& Smprdal, 2006). In addition, MediaWiki provides a discussion page for reflecting on the wiki content. 

2.2 Collaborative Writing 

Collaboration is an activity that enables participants to accomplish a task collectively (Ta-Elhasid & Meishar- 
Tal, 2007; Witney & Smallbone, 201 1). Wikis offer a new way to work collaboratively by creating collective 
content, and as such, they facilitate collaborative writing and group discussion. Collaborative writing is a 
coordinated activity that enables participants to edit and revise each other’s contribution to the wiki task 
(Chao & Lo, 2009; Meishar-Tal & Gorsky, 2010; Trentin, 2009; Witney & Smallbone, 2011), as opposed to 
simply splitting up the task, work independently of each other, and then assemble individual contributions to 
a final wiki. Collaboration is grounded in the social-constructivist learning theory (Vygotsky, 1978), and 
assumes that participants can achieve more in terms of learning benefits than individuals. 

2.3 A Taxonomy to Analyze Students’ Actions Performed on Wikis 

Taxonomies have been proposed in the research literature to classify and analyze collaborative writing 
activities performed on wikis (Meishar-Tal & Gorsky, 2010; Pfeil, Zaphiris, & Ang, 2006). The taxonomy 
used in this paper draws on the one developed by (Pfeil, Zaphiris, & Ang, 2006). This included originally 13 
categories, of which the following 10 were identified as important for this work (Table 1). 

Table 1. Taxonomy and categorization of activities (Pfeil, Zaphiris, & Ang, 2006, pp. 101). 


Category 


Explanation 


Add Information Addition of topic-related information (the information must not consist only of links). 

Add Link Addition of links to an existing set of listed links or linking of a word within an existing sentence 

to a page (links to other Wikipedia pages or to external Internet pages). 

Clarify Information Rewording of existing information without adding new information. Rewording done in order to 
clarify the content (e.g., substitution of certain words for a better understanding, change of the 
word order or deletion/addition of words in order to clarify). 


Delete Information 
Delete Link 

Fix Link 


Deletion of topic-related information (the information must not consist only of links). 

Deletion of links from the set of listed links or removal of the linking function from a word within 
an existing sentence (links to other Wikipedia pages or to external Internet pages). 

Modification of an existing link (can be an alteration of the linked URL or the name of the link). 


Format Contributions that affect the appearance or structure of the whole page (e.g., addition of space 

lines, sorting/moving of paragraphs or links and addition of subtitles in order to structure the 
content). 


Grammar 


Alterations of the grammar (e.g., change of punctuation). 


Spelling Correction of spelling mistakes (e.g., reversed letters or capital letter). 

Style/Typography Contributions that affect the presentation/appearance of the text (e.g., bold/italic/underlined text). 


132 




IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


2.4 Related Work 

Most research work on wiki reports on students’ perceptions of collaborative learning and writing by means 
of qualitative methods. A number of researchers (Arnold et ah, 2009; Britcliffe, & Walker, 2007; Cole 2009; 
Ebner et ah, 2008; Minocha & Thomas, 2007; Karasavvidis, 2010; Lund & Smprdal, 2006) reported that 
students do not collaborate when they use wiki and rarely edit each other’s contributions. Several hypotheses 
were raised to explain the low level of collaboration: limited student contribution, reluctance and resistance 
to use wiki, dominant learning paradigm, problem of ownership, lack of appropriate pedagogy, etc. In 
addition, according to Pifarre and Fisher (2011), there is relatively little research on successful 
implementations of wikis supporting collaborative writing. In quantitative terms, a small but growing number 
of studies have recently drawn on the data log generated by the history function of wikis. Hadjerrouit (2011) 
reported that most students do not collaborate when they use wiki to edit collective documents. Instead, they 
focused mostly on adding content to existing pages and technical aspects. Similarly, Hadjerrouit (2012) 
highlighted the problems and difficulties of using wikis to edit each other’s contributions. Leung and Chu 
(2009) also reported that students worked individually most of the time, and edited each other’s contributions 
if necessary. Likewise, Judd. Kennedy, & Cropper (2010) provided evidence against a general tendency to 
collaborative writing. In some contrast, Meishar-Tal and Gorsky (2010) indicated that adding text was carried 
by a large majority of students, but the percentage of editorial changes was higher than adding sentences, 
because the students were required to edit each other's work. 


3. METHODOLOGY 

3.1 Objective 

This work aims at exploring the extent to which students collaborated to perform wiki projects associated 
with collaborative writing in teacher education. The work focuses particularly on students editing each 
other’s contributions to the wikis and comments posted on the discussion page. 

3.2 Participants 

An eight-week wiki project was introduced in a Web 2.0 technology course. The participants were 16 
students divided into 6 groups of 2 to 4. None of the students were experienced wiki-based collaborative 
writing. Some possessed good technical skills, and some had background in pedagogy. The wiki topics were 
chosen by the students themselves. The specificities and technical features of wikis were introduced to the 
students during the first week of the projects. Lectures on collaborative writing were given in the following 
two weeks. The students were required to submit their wikis for continuous supervision. 

3.3 Learning and Assessment Goals 

To perform collaborative writing using MediaWiki, the teacher provided a set of three learning goals. First, 
the wikis should follow general usability criteria such as technical layout, formatting, and style. Second, the 
wikis must contain information of good quality, without linguistic, grammar, and spelling errors. The content 
should draw on recent curricular development, and include study material that is well structured with heading 
and subheading, images, tables, lists, and references. Third, the wikis should be self-explaining, and offer 
information that is relevant to the target audience. Some of these tasks could be done individually, for 
example adding or deleting content. However, developing an overall wiki requires collaborative work such as 
arguing, discussing, and reflecting on the content through editing each other's contributions, adapting the 
language to the needs of the target audience, designing an overall structure of the wiki, and making 
cross-linking. Given these requirements, the students were encouraged to edit each other’s contributions, and 
take actively part in discussion. Finally, in line with the wiki philosophy based on collaborations, the students 
were not assessed individually, but as a group working collaboratively. Nevertheless, the history function can 
be used to look at the students’ individual contributions to the wiki. 
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3.4 Data Collection and Analysis Methods 

The work used both quantitative and qualitative methods. Firstly, the total number of actions per group and 
category, including their frequencies were collected and analyzed, such as whether the action was an 
addition, deletion or modification of content; addition, deletion, or fixation of a link; formatting, spelling, 
style, or grammar etc. Secondly, the comments raised in the discussion page were categorized by increased 
level of criticality, and analyzed both quantitatively in terms of number of comments, and qualitatively in 
terms of quality of the comments provided. 


4. RESULTS 

This section describes the results achieved in terms of actions carried out on the wiki across the categories 
investigated, and students’ comments posted on the discussion page. Contribution to collaborative writing 
and discussion is also assessed. 

4.1 Contribution to Collaborative Writing 

Table 2 shows the frequency of actions that fell under each of the 10 categories investigated. The total 
number of actions was 2856, which means an average of 178.5 actions per student ( n = 16). Note that a single 
edit may involve several actions, for example a student could add content and delete a link. In this case, both 
actions in the categories add content and delete link were recorded. The table reveals that the most important 
category performed on the wiki in terms of average frequency relative to the total number of actions was 
formatting, followed by addition of content and links, clarification of content and fixing of links, deletion of 
content, style/typography, deletion of links, spelling, and finally grammar. In terms of clarifying content, and 
thus editing each other’s contribution, two subcategories can be distinguished: a student clarifying his/her 
own content, which is not the focus of this work, and clarifying each other’s content, which is the main 
concern of this study. The statistics does not indicate the frequency of each subcategory. However, even if all 
actions associated with clarifying content fell under the second subcategory, which is the best possible 
scenario, the average frequency of 12.04 % cannot be considered as high compared with the percentages 
achieved for formatting, addition of content and links. This is the case of four groups (1, 2, 5, and 6), where 
the average frequency for clarifying content is lower than 12.04 %, and group 4 with a slightly higher 
frequency (14.93 %). The only exception is group 3 that achieved a percentage of 23.79 %, which in itself 
cannot be regarded as very high. Considering these frequencies as the best possible results that can be 
achieved, it can be implied that only a few actions fell under the category clarify content by editing each 
other’s contributions. 


Table 2. Types of actions in each category in terms of frequencies and total number of actions in ascending order. 


Category 

Group 

1 

Group 

2 

Group 

3 

Group 

4 

Group 

5 

Group 

6 

Total no. of 
actions per 
category 

Frequency 
of actions 
in % 

Formatting 

155 

113 

80 

163 

109 

48 

668 

23.39 % 

Add content 

65 

51 

81 

259 

89 

44 

589 

20.62 % 

Add link 

79 

36 

136 

119 

82 

53 

505 

17.68 % 

Clarify content 

24 

21 

114 

138 

44 

3 

344 

12.04% 

Fix link 

69 

8 

20 

68 

39 

17 

221 

7.73 % 

Delete content 

24 

19 

18 

86 

35 

25 

207 

7.25 % 

Style/Typography 

33 

9 

6 

19 

38 

29 

134 

4.69 % 

Delete link 

23 

0 

5 

29 

14 

6 

77 

2.70 % 

Spelling 

13 

7 

8 

29 

6 

5 

68 

2.39 % 

Grammar 

3 

3 

11 

14 

9 

3 

43 

1.51 % 

Total no. of 
actions per group 

448 

260 

479 

924 

465 

233 

2856 

100% 

No. of students 
per group 

2 

3 

4 

2 

3 

2 

n - 16 
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4.2 Contribution to Discussion 

A detailed analysis of the comments posted on the discussion page showed that the students made comments 
on a range of issues. Following (Su & Beaumont, 2010), the content of the comments were analyzed and 
categorized by increased level of criticality (Table 3): 

• Comments on technical issues of the wiki 

• Comments on the wiki content 

• Comments on collaborative writing 


Table 3. Comments posted on the wikis derived from a content analysis. 


Level of 
criticality 

Categories 

No. of comments 
per category 

Frequency 
in % 

Low 

Comments on technical layout, structure, formatting, 
images, tables, lists, paragraphs, headings, subpages. 

106 

55.21 % 

Middle 

Comments on content, proof reading, corrections, 
references, linking, wiki length. 

Comments on reflections related to usefulness of 

74 

38.54 % 

High 

information, critical review of literature, adaption of 
language to target audience, editing each other’ content 

12 

6.25 % 

Total no. of comments 


192 

100% 


Most comments were posted the last two-three weeks before the project deadline. Otherwise it was 
difficult to follow the discussion treads of the respective wikis, because the date of contribution or/and name 
of contributor were missing. As the table shows, most comments were low-level criticality comments 
(55.21%). These related mostly to technical layout, such as insertion and placing of tables and figures, 
formatting of headings and subheadings, etc. Middle-level criticality comments related mostly to addition or 
deletion of content or links, corrections, wiki length, etc. Finally, high-level criticality comments were 
associated with critical issues of the wiki content, such as usefulness of information sources and study 
material used to design the wiki. Only one student emphasized the need to work collaboratively. None of the 
students discussed the issue of how to adapt the language to the characteristics of the target audience. 

There were several comments with low-level criticality, of which the following are representative: 

Now we have nice images and I think there are enough. We can probably insert a picture or two to 
the section on (...). 

Fixed a positioning of images, but I am still not particularly happy (...). Need some more lists to 
meet the requirement. Suggestions would be appreciated. 

The following were typical comments associated with middle criticality: 

We have to at least make sure that none of the links are "empty". We now have about 3000 words 
(...). Thus, 1000 words are still missing. Any suggestions on what we can write more detailed? 

I guess today we will deliver the wiki, so we have to get it done. Who takes care of creating tables? I 
thought I was writing the page about (...), and proofread the entire wiki. We also need to arrange 
the reference list. 

There were few high-level criticality comments. The following illustrates the type of students’ comment: 
The next will probably be that we meet and work collaboratively with the wiki, do you agree? 

Considering that the category with a high-level of criticality is the one that requires most collaboration in 
terms of reflection and critical discussions, it can be asserted that the level of collaboration was rather low. 
This is in accord with the results achieved for the categories of actions carried out on the wikis. In addition, 
the number of comments (192) is very low compared with the number of actions (2856), which means an 
average score of only one comment per 14.87 actions. 
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5. DISCUSSION 

5.1 Summary of Results 

As the results clearly show (Table 2) the most frequent action was formatting, followed by adding content 
and links. Clarifying each other’s content was an action that the students did not carry out much. As a result, 
collaboration among participants was rather low compared with the frequencies of other actions. This is 
confirmed by the comments posted on the discussion page (Table 3). 

5.2 Factors affecting Collaborative Writing 

Although the students were encouraged to edit each other's work, and reflect on their ideas and concerns, the 
results indicate that they preferred to focus more on formatting, technical layout, addition of content, or 
making links to pages than collaboration. In spite of the fact that some issues were discussed face-to-face or 
by means of traditional communication channels, there is little evidence that the students identified gaps in 
their knowledge by reflecting on the content of the wikis by means of literature review and adapting the 
content to the characteristics of the target audience. In addition, most discussions happened the last two-three 
weeks before the project deadline. Likewise, the history function also shows that all group worked much as 
the deadline approached, approximately two weeks before the delivery of the wiki projects. This is in accord 
with previous research indicating the students' tendency to postpone much of their work until just before the 
deadline (ludd, Kennedy, & Cropper, 2010; Leung & Chu, 2009; Meishar-Tal & Gorsky, 2010). This 
behavior somehow undermined the students’ opportunities to fully collaborate, edit each other’s 
contributions, and discuss with their peers. A possible explanation is that the students were more concerned 
about passing the course, while doing as little work as possible than learning collaboratively. The fact that 
they mentioned several times the length of the wiki, the linking of words, uploading of images, insertion of 
tables, and technical layout, is an indication that the students were more concerned about passing the course 
than critically evaluating their knowledge. However, this is not the only explanation for low collaboration. 

Adapting the content to the characteristics of the users was an important requirement, but this issue was 
not raised in the discussion page. This is confirmed by the final wiki products, which failed to thoroughly 
address this issue. Indeed, the analysis of the wikis shows that the students did not identify gaps in their 
knowledge in order to make the wikis more relevant and attractive to the users. Rather, the students focused 
more on selecting content from Wikipedia and other Web sites than reflecting on their knowledge. Adapting 
the content to the characteristics of the target audience is a demanding task that requires knowledge about the 
reliability and usefulness of information sources, including central facts and concepts of the topic, and how to 
connect them to foster meaningful understanding. It is obvious that this issue requires drafting, rephrasing, 
and reworking the language recursively until it fits the needs of the target audience. In addition, an acceptable 
level of language proficiency needs to be taken into account (Li & Zhu, 2011). In turn, drafting and 
reworking the wiki content cannot be done properly without editing each other’s contributions. Another 
explication raised by (Grant, 2009) is that students do not perceive their wikis as an authentic activity that 
requires an authentic audience. They think that their wikis will not continue after the end of the project. As a 
result, combined with the importance of grades, the most important audience for them is their teacher. 

Grant (2009), in line with the results of previous research work (Elgort et al., 2008; Forte & Bruckman, 
2007; Lund & Smprdal, 2006), also suggests that students appear to use practices of individualised written 
work they were accustomed, rather than collaborating to realize shared knowledge, particularly in the 
absence of collaboration models to draw on. The wiki history function seems to confirm the view that 
students tended to approach the writing task more individually than collaboratively by splitting up the wiki 
task into subtasks, not only in the very beginning of the project, which is reasonable and understandable, but 
also throughout the entire project period. This behaviour may be explained by the fact that collaborative 
writing is more challenging in terms of cognitive efforts, active participation, group interactions, and time 
management than just splitting up the wiki task into subtasks, working individually without collaborating, 
and finally putting all the subtasks together to create a final wiki (Hadjerrouit, 2011). 

Another possible explanation may be the assessment form being used (Harsell, 2010; Tetard, Packalen, & 
Patokorpi, 2009). Since the students were not assessed individually, but as a group working collaboratively to 
achieve a common goal, students tended to focus more on the final wiki product than the collaborative 
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process. As a result, the students did not consider editing each other’s contributions as useful or desirable, 
and they preferred more individual work than collaboration. Nevertheless, wikis offer a possibility to assess 
individual contributions by means of the history function, which records all students’ activities, particularly 
when they are required to demonstrate regular contributions and discussion (Grant, 2009, Harsell, 2010). 
However, being aware of the usefulness of the history function to assess individual contributions may not 
automatically facilitate collaboration, if students are not able to develop effective collaborative and 
discussion strategies. 

Finally, usability obstacles of wiki technologies may disrupt the students’ learning experience (Minocha 
& Thomas, 2007). Likewise, the absence of a WYSIWYG editor may prevent students from fully using wiki 
for project collaboration (Chao, 2007). As a consequence, although students are positive when they use 
wikis, they are reluctant to fully collaborate, in accordance with several previous research studies. However, 
for Selwyn (Cited in Grant (2009)) collaboration is less a technological problem than a cultural and 
pedagogical issue. Focusing on removing technological barriers to realizing wiki potentialities is a 
“reductionist thinking equivalent to technological determinism” (Grant, 2009, p. 113). As a result, it appears 
that while wikis might support group work, collaborative writing is not reductible to the technology. Rather 
the role of the teacher, the nature of the task, time management, motivation, assessment, pedagogy, and 
technology integration are crucial elements in encouraging students to work more collaboratively (Caple & 
Bogle, 2011; Lund & Smprdal, 2006, Tay & Allen, 2011). 


6. CONCLUSIONS 

Although this work has its limitations, because of the small sample size (n = 16) and short duration of the 
wiki projects (8 weeks), it can serves as a basis for further explorations in wiki-based collaborative writing. 
Given these considerations, some conclusions can be drawn. Firstly, the results are line with research that 
reports on students’ reluctance to edit each other’s contributions to the wikis. Secondly, the history function 
of the wiki provides an excellent research tool to analyze students’ contributions using an appropriate 
taxonomy of activity categories (Judd. Kennedy, & Cropper, 2010; Meishar-Tal & Gorsky, 2010; Pifarre & 
Fisher, 2011). Finally, factors influencing collaborative writing are identified. These need to be considered by 
teacher educators to successfully exploit the potentialities of wikis to foster collaborative writing. In future 
work, a longitudinal study will be undertaken to explore students’ collaborative writing activities over a 
period of three years to confirm the results of this work. Future research will also be undertaken with a larger 
population of students to strengthen the validity and reliability of the results. 
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ABSTRACT 

One key feature of scientific community is collaborative shared enquiry and problem solving mediated by electronic 
networks. We report on the development and evaluation of an ICT tool to support learning how to learn together (L2L2) 
in Science Education as part of an EC funded project called ‘Metafora’ (FP7-ICT-2009. 4.2/257872). Through literature 
review we isolated some key features of L2L2 and inquiry science processes. We turned these into icons within an ICT 
environment to support planning and reflection of inquiries stimulated by real-world challenges. In this paper we report 
the design-based research study carried out in secondary schools in Spain in order to gain understanding about the 
Metafora’s technological and pedagogical affordances to support students’ awareness of the key aspects of learning 
together and the key scientific inquiiy processes. Our findings suggest that the ICT environment may help rise students’ 
awareness of key collaborative scientific processes. These findings may contribute to the development of tools to support 
more web-mediated collaborative learning. 


KEYWORDS 

Learning to learning together (L2L2), Computer Supported Collaborative Learning (CSCL), Knowledge-age skills. 
Technology Enhanced Learning (TEL), Secondaiy Education. Inquiry-based science learning (IBSE). 


1. INTRODUCTION 

Collaboration is a central tenet of knowledge era. In 2.0 societies, citizens are massively engaged in social 
networking activities to exchange information, collaborate in virtual games and create multimedia artifacts. 
In the economic world, collaborative technologies are mediating knowledge creation processes and project 
management. There can be no doubt that the society requires people to possess the adequate skills to 
participate actively and constructively in collaborative and creative practices (Minocha & Thomas, 2007). 

Although, young people could gain from various kinds of informal computer supported collaborative 
experience, there are no specific web-based tools designed to support them to learn how to learn together in 
subject-specific domains. Furthermore, the current education systems do little to equip children and young 
people with the complex competence of problem solving and learning together with others online. 

The Metafora project 1 funded by the EC Framework 7 ICT program aims to design a pedagogy using 
web-based tools to support learning to learn together in the context of Maths and Science for schools. A key 
technical and pedagogic innovation of the project will be to support learning to learn within a community. To 
gain deep knowledge of any specific domain learners need to integrate learning experiences with shared 
reflection through dialogue in a way that builds conceptual understanding cumulatively. 


1 ‘Metafora’, Learning to learn together: A visual language for social orchestration of educational activities. FP7-ICT-2009.4.2 

Technology Enhanced Learning contract no. 257872 
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2. THE WEB-BASED LEANING ENVIRONMENT: METAFORA 

Metafora aims to provide a holistic environment in which students will collaboratively plan and organize 
their work, as well as collaborate in solving science challenges and problems over a relatively long time 
period. 

A web-based Metafora learning environment is being developed that includes 4 main tools: a) the 
challenge; b) the planning and reflection tool; c) the argumentation tool and d) the microworlds. Next we 
describe each of these tools integrated in the Metafora platform: 

2.1 The Challange 

Challenge-based learning methodology was pioneered by education staff at Apple Inc. and aims to engage 
learners in meaningful learning context; authentic connection with multiple disciplines, multiple points and 
multiple possible solutions and focus on the development of 21st century skills (Johnson & Adams, 2011). 

Metafora project incorporates challenge -based learning objectives and at the beginning of a typical 
Metafora-based activity, a group of students is formed and receives a relatively complex assignment -the 
challenge. The challenge is built in way that will require the students to plan how they are going to approach 
the solution in order to reach it on time. After planning, the group begins with an iterative process entailing 
enactment - discussion - revision of the plan, until the peers obtain a solution for the challenge. 

2.2 The Planning and Reflection Tool 

The planning and reflection implements an understanding of the key features of learning together in a visual 
language intended to facilitate greater awareness of the process of learning together. 

The visual language was designed after a detailed research carried out during the first year of the 
Metafora project. The consortium used literature review and design-based research in order to refine and 
develop a stable visual language for implementation in the Metafora planning and reflection tool (Wegerif et 
al. 2012). The visual language included in the planning and reflection tool has 6 types of components: 

Main activity stages implemented as icons in blue boxes: Explore, Reach agreement. Define 
questions. Build model. Find hypothesis. Test model. Refine model. Draw conclusions. Prepare presentation. 
Reflect on process. Blank (for students to define if required) 

Activity processes that occur within each activity stage implemented as icons in green circles that 
attach to the boxes: Build. Experiment, Hypothesize, Make notes. Propose an alternative. Report, Reach 
agreement. Anticipate, Brainstorm, Evaluate, Gather information. Present, Reflect, Discuss, Simulate, 
Analyze, Allocate roles. Blank (for students to define if required) 

Attitudes implemented as icons in different colored hats: Open, Positive, Critical, Creative, Ethical, 
Rational, Intuitive, Blank (for students to define if required) 

Roles, implemented as icons in grey circles: Manager, Evaluator, Note taker. Blank (for students to 
define if required) 

Resources, implemented as an icon that represent the specific available resource: LASAD and the 
Microworlds. 

Connectors, implemented as black, blue and red arrows indicating any relationship, causal 
relationship and temporal relationship. 

In figure 1 we present the visual appearance of the Metafora platform and the planning and reflection 
tool. 

2.3 The Argumentation Tool - LASAD 

An online dialogue support system using a dynamic concept mapping tool called LASAD was integrated in 
Metafora platform. LASAD is web-based argumentation tool that enables groups of learners to discuss their 
work in a structured way (Loll, Pinkwart, Scheuer, & McLaren, 2009; Scheuer, McLaren, Loll, & Pinkwart, 
2009). LASAD is a collaborative, shared workspace containing a graphical argumentation environment and a 
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chat tool. Students can use this space to share ideas and organize their thoughts in order to solve the 
challenge. 

2.4 Microworlds 


Various microworlds (Kynigos, 2007) which support constructionist learning in mathematical, scientific and 
socio-environmental domains are also integrated in Metafora platform. Students, in order to solve specific 
math and science challenges might use one of these microworlds. 
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Figure 1 . Screenshot of Metafora Platform 


The research study we present in this paper focuses on the implementation and evaluation of the planning 
and reflection tool using a visual language representing the key components and features required for L2L2 
and for shared scientific enquiry 


3. OBJECTIVES AND RESEARCH QUESTIONS 

In our research study we had two main objectives: 

1. To understand and specify the Metafora’s potential affordances to promote the learning and reflection 
about scientific enquire processes. 

2. To study how Metafora’s potential affordances may support students’ development of L2L2 skills. 

This study was conducted as design-based research (Wang & Hannafin, 2005) in which our research 

questions were: 

RQ1: How does the visual language help students to solve the challenge using key scientific processes? 
RQ2: How does the visual language stimulate discussion and reflection about scientific processes? 

RQ3: Does the visual language help students to develop collaborative learning processes? 
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4. METHOD 

4.1 Participants 

Eleven secondary students of year 11 participated in our study. The students were the entire class of a 
Science subject. Students do not work in groups regularly and they were used to embed technology to support 
learning. Students work in 3 groups to solve a challenge-based science project. Students work on the 
challenge during 9 class sessions. 

The Science teacher responsible of this class participated in the design-based research study as a co- 
researcher. 

4.2 Procedure 

Teacher began introducing the challenge and the visual language to the students. She used the interactive 
blackboard. 

The challenge was: 

The water and environmental European committee has fixed in its normative 2000/60/CE that all 
European rivers have to be in good ecological conditions in 2015. 

A study of this European committee realized in 2008 found that Segre River (Lleida, Spain) was in good 
ecological condition only in 75% of its course. The most polluted section of the river is when the river 
crossed the town of Lleida. 

What scientific and rigorous proposals could you think about to influence on the society on solving the 
rivers’ problem. Your ideas and actions might be at different levels: authorities, media, society and 
peers-secondary schools. 

Students were provided with some net resources about main causes that pollute more the river. These 
resources were selected by the Science teacher. However, students could check the open net. 

Afterwards, students planned and solved the challenge using the Metafora planning and reflection tool. 
The pedagogy used during these sessions was: 

• Students worked in small groups during all sessions. 

• Work-in progress presentations and group debate sessions were carried out. During three times 
along the workshop, every group presented their working progress. In this presentation, students were asked 
to present the work done so far but also the group thinking process: reflect and present their discussions, 
problems, how they overcame them, use of visual language, collaboration. . . 

• Final group work presentation and whole class discussion. Every group presented the whole work 
and the group proposal to influence on the society on solving the rivers' problem. 

4.3 Data Collection 

• We collected all students’ group work realized on the computer and students’ group discussion 
during small group work were video-audio-recording using video recorder programme -CAMSTUDIO. 

• Video recording sessions of work-in progress presentation and final presentation. 

• Video recording of students’ dialogue while working together. For the purposes of this paper, we 
have transcribed and analyzed the dialogue of one group. 


5. FINDINGS 

5.1 How does the Visual Language Help Students to Solve the Challenge Using 
Key Scientific Processes? 

To answer this research question we analyzed the small group work in the planning tool and their work-in 
progress presentations to the whole group class -in which students present what they did. for which purposes. 
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how they were thinking in order to better solve the challenge. All the groups organized their challenge 
resolution process around the “activity stage” icons which represented a scientific objective to solve the 
challenge. 

Analyzing the planning and the icons used by the three groups, we observed students took in 
consideration the next five scientific enquiry stages: 

• Define the problem 

• Hypothesis 

• Methodology - Experimental design 

• Analysis of results - discussion 

• Conclusions and proposals to solve the challenge 

These findings show that Metafora planning and reflection tool supported students’ creation of an enquire 
process because students establish the main scientific enquire stages highlighted in the literature (e.g. 
Hakkarien, 2010; Shimoda, White & Frederiksen, 2002). 

Besides, students used the “activity processes” to unpack the processes and actions of the scientific 
activity stages. The use of the “activity processes” helped students to better define and fulfill the scientific 
objectives of each activity stages. 

Furthermore, the analyses of the data showed that “activity processes” icons were mainly used for the 
next three purposes: a) as an aid to start thinking in possible actions: brain storming; b) to reflect about what 
they did and consequently plan the next step to solve the challenge and c) to organize and structure their 
actions. Next we present three excerpts of the dialogue students showed in their discussions about what to do 
and how the visual language mediate in their scientific discussions in order to solve the challenge. 

Next we will present an example of the use of the “activity processes” visual icons as an aid to start 
thinking in possible actions: brain storming. 

We all agree we have to start defining the problem. But., what next ? Let’s see what actions we can 
do [[student drags some process icons to the screen and all the members started a discussion 
about what processes should follow]] 

An example of the use of the “activity processes” visual icons as an aid to reflect about what they did and 

plan next step is presented bellow: 

Ok, let’s see, previous knowledge, and then we observed the data, explored the cartography link 
and the water agency link, and then we researched for new information. 

...But we don’t have enough I think now we have to obtain new data about the river: look at this 
map [[open a link from the web resources]] it’s clickable! It shows the quantity of water of the 
river at different points. How much water does it have in the different stages of the river ? and in 
Lleida? Look We can compare them. 

An example of the use of the “activity processes” visual icons as an aid to organize and structure their 
actions is presented next: 

Ada: I would put all of this in one block: reflect and analyze. All the information we have in here ... 

Thus, all this information [[pointing at text written in one of the boxes]] is the information we got reading 
on the web. 

Alan: Yes 

Ada: I will put the icons reflect and analyse, because we have already analysed it, haven ’t we? 

Alan: wait, wait, say it again and I will put the icons 

Ada: I try to say what we are doing now? 

Alan: Yes, and I agree [[she looks for an icon and drug to the computer screen ] ] 

Ada: Brain storming [[this is the icon that Ain druged]] no, no, this later. We have done the analyses... 

Observation of the planning process combined with feedback from students’ in-progress presentations 
suggests that “activity stages” and "activity processes” visual icons promoted students to consider aspects of 
the scientific research process that they would not thought otherwise. Therefore, the visual language included 
in the planning tool enrich students’ scientific enquire processes. 
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5.2 How does the Visual Language Stimulate Discussion and Reflection about 
Scientific Processes? 

We transcript and analyzed the dialogue of students of one group of students and in one class session. First, 
we track in the transcription for words related with the visual language. During this session students 
intensively used the words of the visual language in their discussion. In this line, students used 30 times 
words related with the “activity stages” but they only put one icon of this category in their planning map. 
Students used in their discussion words such as: conceptualize the challenge, methodology, predict the 
results, hypothesis, and steps to follow. 

Regarding to the visual language referred to “activity processes”, students also included intensively 
during their group discussion words related to processes as: analyze, observe, brainstorm, explore, search for 
new information, discuss. 

From our point of view, this finding is very important because it might confirm that the visual language 
had a positive impact on students’ dialogue and on the way students organize their science thinking. 

In future research studies, we are intended to use of “text analyses software” as “Wordsmith tools” to 
better analyse the use and the impact of visual language on the learning of scientific enquiry processes. 

Additionally, a deeper analysis of students' dialogue showed the presence of students’ reflection about the 
most appropriate scientific processes to carry out in order to solve the science challenge. In this dialogue it 
can be seen how Metafora visual language promoted and mediated the reflection about scientific process to 
solve the task. 

Ada: let’s see. Then, when we do that then? 

Alan: so, in theory we are still here. We have not done anything, right? ((laughs)) 

Ada: yeh ... but from this, we should do an experimental design shouldn ’t we? Or something. 

Alan: if 

Ada: This is experimental design, right? [[Looking for experimental design icon]] 

Alan: wait, wait, and wait. First is the hypothesis 

Ada: We put and if we do better steps to follow first, and obser\’e second 
Anna: if ... and reflect as well. Now we are reflecting, aren’t we? 

Ain, if also 
Ada: thinking... 

Alan: here and to reflect put an arrow. So, after everything we've done we look in the mirror. Can 1 
do it? ... [[Ask for the photocopies of the icons]] 

[[Ada recorded in the Freestyler put icons on the last day]] 

5.3 Does the Visual Language Help Students To Develop Group Learning 
Processes? 

In collaborative learning situations, the process of shared meaning making is seen as just as important as the 
actual outcome of the activity. In this respect, Mercer and Littleton (2007:25) argue that collaboration 
involves “a co-ordinated joint commitment to a shared goal, reciprocity, mutuality and the continual 
(re)negotiation of meaning.” 

A key concept, related to this idea is the concept of ‘intersubjectivity’, which signifies the process of 
developing communality in joint activity. Linell (1998:225) argues that, for collaborative projects to be 
successful and truly collaborative, all parties must be ‘mutually other-oriented’. Additionaly, in the context of 
Computer-Supported Collaborative Learning, Wegerif (2007:181) claimed that it is necessary to develop, 
through social interaction, a “dialogic space”, which he sees as the social realm of the activity within which 
people can think and act collectively, thus opening up a space between people in which creative thought and 
reflection can occur. 

In this section we wondered if the Metafora planning and reflection tool stimulated and mediated the 
development of key collaborative processes. 

The analyses of the session we transcript showed students shared meaning making, took reciprocal 
perspective, students were mutually other-orientated and students created a dialogic space in which they 
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thought and acted collectively. Next we present an excerpt in which collaborative learning processes are 
explicit. 

The context of the next extract is: students are discussing the different concentration of nitrites and 
phosphates in the water in different points of the river. They are analyzing different graphics from a web 
resource. 

Ada: that’s strange... However, I still do not understand why during the watering season there is 
less [referring nitrites]. Maybe because they are more dissolved. I do not know. 

Anna: I suppose, because of those about the fields, and how many times you can water the fields, 
right? 

Ada: yes 

Anna: You have to water the fields every 15 days, ok? When you do not have to water is because 
the humid is high. 

Ada: then, during the watering season, there is less water because the plants absorb it? 

Anna: yes. Because the land absorb it. They have that... 

Ada: Likewise. So the land, during the watering season absorbs water and in the water is where 
are the phosphates and nitrites, so is logical that there are less... and just when there is no 
watering ... land does not absorb the water and then the water would pass without any difficulty 
and go to the river again. 

Alan: good explanation, different to my one... but yes, what you have said is also possible. 

Anna: I know this because my uncle has afield, and 1 know that he waters every 15 days, and for 4 
to 5 hours, they put water in the field till the whole field is watered 
Alan: yell... it can be, can be 
Ada: yell, then we can base on this. 

Ain: Ok 

Ada: with what you are saying DNLA. It is true. 

Alan: so if there is no watering, they are not fixed in the land [referring to fertilize] and they go to 
the river. 

Ada: yes 

In this extract students are making explicit their own ideas and arguments and provide reasons, 
justifications, warrants and evidences to support one’s opinions. Ada started the dialogue explicitly 
expressing a doubt. Anna tried to explain the issue, giving arguments and reasons. Students showed an 
explicit effort to try to construct common knowledge which would enable them to make a conclusion. In 
doing so, students re-elaborate their own and other's ideas and reasons. 

Students, in this extract, made the effort to provide different kind of arguments, for example, Alan gave 
evidence that came from his own experience (his uncle). Besides, students made the effort to agree in the 
conclusion. 


6. CONCLUSIONS 

This paper discusses the affordances of a new learning environment, supported by new technology that is 
currently under development -Metafora. Learning how to learn together (L2L2) in science is a key complex 
skill or competence for knowledge age work. The Metafora project aims at developing a better understanding 
of this complex skill through specifying key features of learning together science processes that students need 
to be aware of and able to work with, and by embodying these features in a visual language which forms the 
main component of a planning and reflection tool. 

We have reported a design-based research study in which the main objectives were to understand and 
specify the Metafora’s potential affordances in promoting the learning and reflection about scientific enquire 
processes and in supporting students’ development of L2L2 skills. 

Findings suggest that the visual language we have developed can help raise students’ awareness of key 
collaborative scientific enquiry processes. The Metafora visual language helped students to unpack and 
reflect about the scientific processes to solve a complex science challenge. Additionally, the Metafora visual 
language promoted students’ awareness about aspects and components of their collaborative learning 
processes in science. 
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The development of this visual language and its initial successful trials has potential pedagogy 
significance in Science Education. In our study, the tool has shown itself to be of value to science teachers 
who need to teach not only the content of science but also the process of scientific enquiry. Students of our 
study reported that Metafora helped them to reflect about the nature of scientific methodology and about 
scientific inquiry processes followed by the group. The Metafora planning tool allows the representation of a 
shared inquiry process. This representation helped students to better understand the scientific methodology 
and how to apply it in a specific context. 

However, further research is needed to investigate the impact of using this tool on the ability of students 
to learn together with others in new situations. This web-based support for groups learning to leant together 
has to prove its significance in more disciplinary fields, across educational contexts and with larger studies. 
Further research is already planned to explore the potential of the Metafora planning and reflection tool to 
support distributed individuals learning together via the web. 
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ABSTRACT 

Children nowadays consume and manage lots of interactive digital software. This makes it more interesting and powerful 
to use digital technologies and videogames supporting learning experiences. However, in general, current digital 
proposals lack of in-situ social interaction supporting natural exchange and discussion of ideas in the course of creative 
learning activities. Moreover, learning activities based on games should put the emphasis on the phase of creating 
artifacts for the game ecosystem, not on just playing or consuming contents. With the aim of supporting learning 
activities focused on “playing to create games”, this paper presents the implemented middleware to support this process 
from a technical point of view. The system allows the creation on an interactive tabletop of both single and advanced 
entities composed of shapes that can be enacted according to physics principles and rule-based behavior to create game 
ecosystems for learning purposes. 


KEYWORDS 

Game, Learning, Tabletop. Collaboration. Entity, Middleware. 


1. INTRODUCTION 

Nowadays children and teenagers have access to lots of interactive digital media and software in many 
different forms by means of several types of devices. This continuous contact with technologies contributes 
to digital literacy and makes them advanced citizens in the digital age. As a consequence, these users will be 
the base of skilled users in the future who will be able to deal with more complex and advanced software than 
ever. 

Digital media and software should be used as a help supporting learning in combination with traditional 
materials rather than using the software as an exclusive mean for learning, as reported by McFarlane 
(McFarlane, 2002) or as the pedagogical model elaborated in the work by Gros for formal learning settings 
(Gros, 2007). In particular, these authors involved commercial videogames taking advantages of the 
knowledge that students already have on technologies as digital natives. Videogames are useful in this 
respect because they also maintain both motivation and engagement usually high (Michael, 2005), which is 
important for learning activities. In addition, they explain that using commercial video-games instead of 
software to conduct specific tasks is advantageous for several reasons: they are cheaper than designing and 
ordering the development of specific digital serious games; and in spite of the low customization capabilities 
of the commercial video-games, given that the pedagogical model can slightly change from one teacher to 
another and from one issue to other, the wider range of commercial video-games available gives flexibility in 
the purpose of enriching the learning setting. All this means that teachers demand platforms that permit 
activity customization for the development of learning sessions rather than a video-game to support already 
pre-established learning activities. 

This vision is not essentially new, although videogames had not been applied in a generic way for 
learning purposes until these studies in real classroom settings. Before the digital age, Clark Abt already 
wrote about serious games as tools for learning and the way they should be used (Abt, 1970). Basically, he 
considered that games are useful tools for learning when they are played and consumed. However his major 
contribution is the idea that they are more powerful when teachers and tutors consider playing to create 
games as the primary learning activity and involve students in the process. In his opinion it is therefore more 
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rewarding from a learning point of view because this vision entails design tasks, making up rules, and 
producing the materials and the logics behind the game to be created co-operatively rather than just 
understanding pre-established game rules. 

The perspectives illustrated above suggest that it will be more useful for teachers to have flexible digital 
tools available at hand that may be integrated in their respective pedagogical models for formal instruction; 
and that creative and social processes as the ones considered before the digital age along with higher 
motivation provided by the use of technology are of special interest for learning. Thus, our future long-term 
purpose is to provide a framework with these characteristics to be used in learning activities. Teachers would 
be able to create incomplete game ecosystems and design several creative learning exercises around them. 
These exercises would be addressed by the students in small groups, interacting and collaborating around the 
table. The creation of artifacts to complete the ecosystems would support the kind of authorship proposed by 
Abt, whereas the resulting game ecosystem could be enacted to test whether it works as expected, triggering 
reflection and discussion processes for refining the created artifacts. 

The main contribution of this paper is the middleware that has been implemented to support such a 
framework from a technical point of view. This middleware allows the creation and enactment of virtual 
ecosystems on interactive tabletops composed of 2D entities. An example about a vintage game ecosystem is 
presented to illustrate and test the basic middleware functionality. On the implemented infrastructure several 
learning activities can be prepared, ranging from storytelling performances to programming physics 
simulation. 

This paper is organized as follow. Section 2 describes some related work, focusing on those supporting 
some kind of artifact creation by children or teenagers. Section 3 introduces the entity-based model to 
support the construction of game ecosystems. Section 4 describes the middleware architecture implemented 
to support the enactment of game ecosystems. Section 5 describes one of the sample ecosystems that has 
been created to test the implemented middleware. Section 6 explains the associated rationale to use the 
platform for supporting learning activities. Finally, this paper concludes by summarizing the contribution of 
this paper. 


2. RELATED WORK 

Although there are a variety of proposals presenting software systems to create game ecosystems, in this 
section we are interested in works that are more focused on the creation of some kind of entities for learning 
activities. LogoBlocks is a graphical programming language to support programming for the LEGO 
programmable brick (Begel, 1996). The language uses a drag&drop methaphor in a Windows-Icons-Mouse- 
Pointers (WIMP) user interface. The brick is a small computer that can be embedded and used in LEGO 
creations by reading from sensors and controlling engine activations. In this way, children and teenagers can 
create ecosystems where robots are physical programmable entities. 

Scratch is a graphical programming environment that allows children to program interactive stories, 
games and animations composed of a set of sprite-based objects (Maloney, 2004). The programming 
language to specify behavior is also based on a drag&drop metaphor of virtual blocks representing 
instructions. The main screen of the tool shows the stage and the sprite representing the entities, allowing 
program debugging, and testing new ideas increasingly and iteratively. The environment is a single-user 
application based on WIMP interaction. 

Another interesting work is Agentsheets (Repenning, 2000). It is a tool based on agents that allows users 
to create simulations and interactive games. Users can create simulations of sprite-based agents in a 2D world 
arranged in a rectangular array. The users are responsible for designing the visual aspect of agents by 
drawing icons, so that these agents are actually sprite-based entities. Their behavior is based on event-based 
rules which are edited following a visual approach to the rewriting rule paradigm. 

Topobo is a 3D constructive assembly system that allows the creation of biomorphic forms like animals 
and skeletons (Parkes, 2008). This is achieved by means of pieces embedded with kinetic memory. Topobo is 
designed to be a user interface that encourages creativity, discovery and learning through active 
experimentation with the system. It can help students to learn about several educational concepts on physics 
such as balance, center of mass, coordination and relative motion. 
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Finally, another interesting work is ShadowStory (Lu, 2011). It is a storytelling system inspired in 
Chinese traditional shadow puppetry. Children use a Tablet PC to create digital animated characters, and then 
they are allowed to perform stories on a back-illuminated screen, controlling the characters with simple 
movements by means of orientation handheld sensors. Thus, to some extent the activities considered in this 
system would support the main aspect of the Abt’s ideas since children are allowed to create the entities and 
later use them to perform a public story intended as a learning task. 

As seen in the previously described related work, there are many different technologies used to support 
the idea of creating some sort of games or interactive media. The difference with regards to our approach is 
primarily the social component provided by tabletop technology. It supports more balanced interaction in 
smalls groups and natural interaction with fingers and hands. Moreover, the combination of mechanisms for 
simulating and enacting ecosystems has not been considered in any of the other approaches. Our work unites 
the physics-based simulation along with logics behavior based on rules. It provides a higher level of 
customization and possibilities for a wider range of activities. 


3. PHYSICALLY-BASED ENTITY MODEL 

Aiming at an environment in which the main tasks would focus on the creation of game ecosystems, it is 
important to consider the factors that will determine the primary editing units of such ecosystems. The 
ground technology being considered is an interactive tabletop. In contrast to other existing technologies, we 
have chosen this one because we are interested in fostering collaboration and discussion during the task of 
creating the game. This social aspect is relevant as it enhances the learning process by exchanging and 
refining ideas between participants. Because this technology supports easily and effectively the sharing of 
digital objects on the flat surface (Hornecker, 2008), the characteristics of the game ecosystems to be 
elaborated should take advantage of this. Thus the game ecosystem should consider the composition of parts, 
being carried out collaboratively, and the interaction techniques should be easy to perform following typical 
gestures such as tapping, dragging fingers, etc. Specifically, we have considered a game ecosystem to be a 
surface where 2D virtual entities can be located and simulated according to intelligible physics (i.e. 
kinematics). 

In addition, we have taken two design decisions with respect to the form of the entities that will determine 
the capabilities of the model required and the range of alternatives for learning activities in the future. Firstly, 
we would like to be able to include concepts such as the instantiation of entities. It would allow creating 
instances at runtime under some conditions, reusing preexisting definitions and dealing with abstractions. It 
would be what classes are for OO programming languages. These elements are useful to develop the 
computational thinking skill (Wing, 2006), considered as a very important one for personal development. 

Secondly, although entities could be single bitmaps or sprites, and this would be probably enough to build 
most 2D game ecosystems, we are interested in allowing also the definition of entities composed of 
architectural units in the same way that a skeleton is composed of bones. In this way, we could also support 
the creation of more advanced behaviors and the more interesting learning scenarios based on storytelling 
containing puppets. 

To fulfill these goals we have defined a meta-model to cope with the visual representation and physical 
properties of entities (see Figure 1). Basically, the stage or game ecosystem is composed of embodied 
entities. An embodied entity-type ( EmbodiedEntityType ) is physically defined in terms of its internal 
structure ( StructureType ) and the protocostume ( ProtoCostume ) covering such a structure. The structure can 
be composed of a single component ( StructuralComponent ) or several components put together by joints 
(Structural Joints) to create more complex structures. These components are simply covered by a visual 
representation by means of images ( ProtoSkin ). Figure 2 shows several examples of entities that can be built 
using the meta-model. Essentially, the structure-type is like the internal skeleton that determines the physical 
simulation whereas the proto-costume is like costumes or the dresses of the structures used for visual 
representation purposes. From these elements, entities are simply instantiated embodied entity-types. 
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Figure 1. Meta-model for entity types. 


RectangleStructuralComponent 



Figure 2. Entity types examples. 


4. ARCHITECTURE AND ENACTMENT MODEL 

The designed architecture leading the implementation of the middleware to process and enact ecosystems 
defined in terms of the meta-model described above is explained in this section. There are three layers in 
which the middleware can be broken down (see Figure 3). The Model layer refers to the definition and 
specification of the game concepts and their storage. This layer is responsible for managing the concepts 
presented in the previous section concerning the stages, entities, rules, etc. available in the game ecosystem 
being created, by giving support to update and retrieve all these basic elements. 

The Controller layer holds the core functionality to orchestrate the simulation of a world stage. Basically 
the simulator has to take a stage to be simulated from the ecosystem, and all the data from the Model layer. 
The simulator has an event queue for the event occurrences produced during the simulation. Three types of 
events are queued: those thrown by the actions when executed in this layer, those being consequence of the 
physical simulation carried out by the underlying physics engine; and those related to the gestures or 
interactions of the user on the surface. This queue is regularly consumed by the rule processor, which 
determines which rule must be triggered, and eventually performs the execution of the action of the matched 
rules. Actions can involve changes in logical entity properties (e.g. increment the variable “Hits” of a block 
in a Breakout game) or in visual properties of the game (e.g. change of the visual representation or skin of an 
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entity). The Controller layer uses the services from the Model and View layer to address these two types of 
run-time changes respectively. In this way, the simulator controls the evolution of the stage simulation by 
consuming events and invoking services on the model or view as needed. 

Finally, the View layer is responsible for visualizing the representation of entities under simulation. It 
offers a core set of view services that allow us to include entities and change their visual properties from the 
Controller layer. 

The View layer is by far the most complex layer to implement. In order to perform the visualization of 
physics components, this layer relies on the Farseer physics engine 1 . It is an open-source physics engine 
simulator that allows the simulation of shapes defined internally in terms of bodies, geometries and joints. 
Since Farseer does not understand the concepts in which the visual structure of entities is defined (i.e. 
StructuralComponents and Structural Joints), the View layer has to translate the components of the entities in 
our model into Farseer primitives. Moreover, it also has to maintain the correspondence between these 
elements in Farseer and the model in order to be able to track which entities are eventually producing 
physical events such as collisions. In this way, when two shapes collide, the middleware is able to determine 
which entities these shapes belong to, and therefore it is able to throw a physics event occurrence associated 
to the involved entities that will be queued in the Controller layer for further processing. All the components 
of this middleware have been implemented in C# and using the Microsoft Surface SDK. 
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Figure 3. Architectural logical view of the implemented middleware. 


5. MIDDLEWARE VALIDATION: A BREAKOUT GAME 

The middleware allows the instantiation of any game ecosystem defined in terms of the model. Tests are 
necessary to prove that the concepts and the definitions present in the meta-model along with the 
corresponding implemented operationalization are functionally correct. To carry out this kind of validation, 
vintage game ecosystems seem good candidates for the test because they are well understood and provide a 
fair level of complexity. In particular, a basic Breakout game 2 ecosystem is presented here. 

In the original game, there are some layers of bricks filling the upper side of the screen, a paddle in the 
lower side and bouncing ball. The user has to move left-right the paddle to prevent the ball reaching the 
lower side of the screen, and he/she has to get points by breaking bricks when the ball hits them. 

For the present use case, we have considered three extended features with respect to the original game: 
bricks have a longer life, represented by an integer property “Hits”, which counts how many times the brick 


1 Farseer Physics Engine in Codeplex: http://farseerphysics.codeplex.com/ 

2 Breakout Videogame: http://en.wikipedia.org/wiki/Breakout_%28video_game%29 
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can be hit until destruction; there are two paddles whose scope is limited to a half of the stage intended for 
two players; and two players are present, who get points up in their scoreboards when they break bricks after 
hitting the ball with their own paddle. Figure 4 shows a picture of a user playing the game ecosystem being 
enacted. 



Figure 4. User playing the game ecosystem being enacted by the middleware. 

The use of this game as a testing example is suitable for several reasons. Firstly, it is simple enough to 
perform a meaningful proof. Secondly, it requires multiple instances of the same type (e.g. several bricks) 
which could have different property values (e.g. one brick could require 2 hits in order to be destroyed 
whereas others could require only 1 hit). Thirdly, the type of rules required to control the evolution of the 
game are simple enough as to be intelligible and explained. Overall, it requires that the orchestration 
performed by the Controller layer to work properly, combining the physical events (e.g. event Collision 
between ball and brick) and logical events (e.g. the property Hits of a brick reaches 0, and as a result it will 
have to be destroyed and removed from the on-going simulation). 

The screen has been modeled as a stage in our middleware. The main entities to be simulated are the 
bricks, the paddle and the ball. In addition, displays for the scoreboards are defined. All of them required 
defining the corresponding structure-type and the protocostumes in the Model layer, which specify the 
physics behavior and the visual representation of the entities respectively. Erro! A origem da referenda 
nao foi encontrada. illustrates graphically these concepts from the model. Note that two protocostumes have 
been defined for bricks since we would like to represent the bricks in a different way according to the value 
of the Hits property (e.g. in red those whose Hits property value is set to 1 and green when it is set to 2). The 
definition of types, structures and costumes provides the specification of the look and the behavior in terms 
of physics. Moreover some rules have been defined in the game ecosystem, which represent the logics of 
control governing the gameplay. Although it is not possible to show here all the required rules for space 
limitation. Figure 6 shows two sample rules defined in the ecosystem. They illustrate the expressiveness 
supported by the rule language implemented in the middleware. On the one hand. Figure 6(a) specifies a rule 
that establishes to 1 the property LastPaddleHit of the Ball when the Paddle 1 collides against the Ball. It will 
be useful to know which scoreboard has to be incremented when the ball hits a brick. On the other hand. 
Figure 6(b) specifies that the property Hits of the brick collided by the Ball has to be decreased when the 
collision occurs. 


EmbodiedEntityType StructureType ProtoCostume 


BallType 


BrickType 


PaddleType 



Figure 5. Breakout game modeling. 
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PRIORITY: 10 

IF S: Paddlel THROWS an EVENT E: Collision 

AND E.SIaveEntity = Ball 

THEN WITH T: Ball 

PERFORM 0: T.LastPaddleHit = 1 

(a) 


PRIORITY: 15 

IF S: ANY BrickType THROWS an EVENT E: Collision 
AND S. Hits > 0 AND E.SIaveEntity=Ball 
THEN WITH T: ANY BrickType 
SO THAT E. Entity = T. Value 
PERFORM 0: T.Hits = T.Hits -1 
(b) 


Figure 6. Sample rules. 


6. ENHANCING LEARNING: PLAYING TO CREATE GAMES 

The implemented middleware presented in this paper allows the creation of both simple and complex entities 
enacting them in the world ecosystem according to physics principles. Since the system supports not only the 
creation of game ecosystems but also the creation and customization of partial ecosystems to be completed, 
this platform facilitates a range of activities that can be proposed to enhance learning, following the ideas 
discussed by Abt and Gros in their respective works and leading to activities under the vision of “playing to 
create games”. It provides enough flexibility for teachers to design sessions and decide which parts should be 
already provided and which ones should be created or completed by pupils. 

The activities can be oriented towards several core tasks. For example, the system can be used to create 
characters for storytelling purposes so that the surface is the stage and anchorage points for entities and 
gravity are defined so that entities resemble like flattened puppets to perform the story. Another possible 
scenario can be focused on teaching essential physics by creating single entities as punctual masses aiming at 
some specific goal and then exploring how the system evolves. 

In addition, more complex and complete ecosystems can be devised by including reactive behavior 
expressed by means of rules, as shown in the Breakout game example. Besides the specification of the visual 
aspect, the rule behavior can be edited by means of a rule editing tool invoked from the world editor. The 
implemented editor follows a set of drag& drop interaction techniques for some part of the rules combined 
with data-flow expressiveness. Data-flows allow users to edit expressions collaboratively using their fingers 
and some tangible objects like pucks for input on the surface without using any keyboard or mouse 
peripherals. Each task to complete the definition of a game ecosystem is oriented as a collaborative learning 
activity. 

To demonstrate the flexibility and usefulness of the system for creativity learning tasks, the system has 
been used in two experiments involving teenagers. In these experiments subjects faced problems such as 
creating articulated entities (Catala, 2012a) and functional Rube-Goldberg machines (Catala, 2012b). The 
activities were considered within a discussion-action-reflection loop so that students could create the 
proposals collaboratively and interact between them. The tests showed that tabletop technology maintains 
students motivated, helps in sharing digital objects more effectively and that the aforementioned loop 
facilitates fairer co-operation interaction patterns, which are positive in terms of social skills development. 
Figure 7 shows two subjects interacting to create a structure for their puppet-like entity. 



Figure 7. Two users creating an entity. 
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7. CONCLUSION 

On the idea of providing a platform for learning activities under the vision of “playing to create games”, this 
paper describes the implemented middleware to support such a learning framework from a technical point of 
view. It relies on a meta-model for physically-based 2D entities that can be enacted according to physics 
principles and rule-based behavior. The system uses an interactive tabletop as ground technology, which 
provides several advantages over other existing approaches. It does not only keep motivation in high levels 
but also encourages social participation and enables more natural interaction in collaborative tasks. 

Our proposal focuses on the creation of several artifacts to create game ecosystems. It is flexible since it 
provides a desirable level of customization for teachers, so that they can design the learning sessions as 
needed. In this sense, the combination of two enactment principles (physics and rules) introduces even more 
possibilities for designing a wider range of activities than those proposals that only use either programmable 
simulations or storytelling live performances. 

There are also some clear limitations. On the one hand, the use of the tabletop is not for masses but 
intended for small groups, so that the discussion and interaction can be properly conducted from a 
pedagogical viewpoint. On the other hand, the type of game ecosystems is based only on 2D performances 
and simulations, leaving out more advanced interesting scenarios based on 3D concepts. Nevertheless, it is 
still a powerful tool for teachers as it allows them to create partial ecosystems for many different learning 
scenarios adapted to their needs as discussed in the paper. 
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ABSTRACT 

Many alternatives exist for setting the delivery, content, direction, tone, and priorities for a Critical Inquiry/Thinking 
general education program. Review of our university's overall general education program indicated the need, and 
overwhelming faculty approval, for a program to improve critical thinking skills, to specifically include exploratory 
learning and improved cognitive abilities. As we moved ahead we formed a Critical Inquiry Committee for in depth, and 
participative planning. One of the preliminary, investigative steps was to survey peer institution critical inquiry 
programs. The results of the study are reported below, along with our steps in developing our first year, one credit, 
general education critical inquiry program. Our paper includes details of our faculty development program and the results 
of our pilot program implementation. Our full program is scheduled for implementation during fall 2012. 


KEYWORDS 

Critical inquiry/thinking, exploratory learning, analytic thinking, cognition, general education, curriculum 


1. INTRODUCTION ~ IS THAT GOING TO BE ON THE TEST? 

Faculty members are aware of the importance of critical inquiry (Cl) and critical thinking CT), however, 
these skills are seldom specifically addressed directly in courses. Further, critical thinking is generally 
assumed to be a byproduct of secondary education, rather than an end in itself. While critical thinking skills, 
exploratory learning methods, and cognitive abilities are considered essential to higher education success, 
and prerequisite to satisfactory course progression, these skills are not often directly addressed either before 
or during college. Even in the more objective and quantitative disciplines, such as math and science, the 
inquiry, exploration, questioning, and decision processes (critical inquiry) are ordinarily secondary to 
achieving the “correct” answers. (Sadly, with the recent, absolute prioritization of K-12 test scores, learning 
to think may well be even more sidelined in favor of “answer-driven test outcomes.”) In higher education, 
the trend continues. Students learn, all too frequently, what to do, rather than why they are doing it. We feed 
the “is that going to be on the test mentality.” In spite of all the pedagogical discourse to the contrary, 
students continue to learn their outcomes, learning them in the traditional, non-integrated “silos” of teaching 
disciplines, seeking the test answers — rather than the questions. Change is difficult. 

Over the last decade, the American education process, especially higher education, has focused on content 
areas of concentration that emphasize technical and memory based skills ( Arumand and Roksa). The absence 
of training on critical thinking, along with analytical ability, is ever present. Emphasis on test scores for both 
undergraduate and graduate admissions are at an all time high. In fact, educators and institutions are 
evaluated on test scores of both incoming and graduating students (Resmovitz). The idea of teaching critical 
thinking has gained popularity in recent years (Mulnix). It was often believed that students were exposed to 
critical thinking philosophies during their primary and secondary education programs. 

Notwithstanding faculty members repeated complaints that students are not adequately prepared for 
success in their courses, little attention is paid to the skill of thinking, Socratic dialogue, or exploratory 
learning. Because we cannot readily change the st5udent preparedness, the most logical approach should be 
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to consider change in our own pedagogical approaches. For example, Clabaugh, Forbes, and Clabaugh 
suggest that students will demonstrate greater gains in critical thinking skills when the learning objectives for 
specific courses are specifically developed toward building these skill; perhaps obvious, yes - but is it done? 
It may be that small steps, specific activities and assignments directed solely at cognitive development can 
combine to make significant progress to improve thinking and exploratory skills. More pervasive 
alternatives, entire programs focused solely on critical inquiry, represent are another avenue to effect 
improvement. 

Our university, a small undergraduate institution in the Southeastern United States, determined that the 
development of Cl skills for entering freshmen should be the cornerstone of our Quality Enhancement Plan 
(QEP), which was undertaken in preparation for a forthcoming accreditation review. We felt that a 
successful critical inquiry course would enhance student thinking skills and their interest in inquiry and 
learning. A welcome accompanying plus was that faculty involved in the Cl program would naturally 
advance their own Cl skills, and perhaps more importantly, their ability to bring Cl skill development to their 
existing, traditional classes. Win, win... Creating a solid faculty development program to prepare 
[eventually all] faculty to teach Cl courses became the cornerstone of our Cl implementation strategy. Better 
trained Cl faculty, across all disciplines would foster and cultivate continued critical inquiry and thinking 
skills throughout the all levels of our curriculum. 

There are many options to set the direction, content, tone, priorities, and activities for a viable Cl 
program. This can range from a dedicated Cl course, sequential courses, discipline embedded Cl segments, 
integrated cross-discipline approaches, and step approaches. Following the Cl model, with exploration as the 
key foundational step, we felt learning about the types of Cl programs in use would be valuable in designing 
our program. This would guide us in discerning among programs to find the greatest likelihood of success. 
In connection with the planning and implementation of our QEP, we decided to survey Cl practices at peer 
institutions, including the perceptions of administrators on successful Cl efforts. The purpose of our study 
was to: (1) determine the types of Cl programs currently offered, (2) find out how Cl is addressed within 
various curricula, and (3) understand the sometimes conflicting views on which types of programs are likely 
to have the greatest success. Our study gives a summary of Cl practices at other institutions, presents our 
experiences in creating our own Cl program, including the pivotal Cl faculty development program, and 
concludes with lessons learned as we implemented our pilot program in 2011-12. 

2. CRITICAL INQUIRY, SURVEY & COURSE IMPLEMENTATION 

Formal and informal faculty/student interactions reinforce retention and academic performance; this is 
especially important in the first year of college. Faculty teaching gateway courses (e.g., English, math, etc.) 
if trained more systematically in Cl methods hold a key position to make headway in critical thinking skills. 
This should have synergistic advantages in furthering the Cl agenda, at the same time improving delivery, 
understanding, and retention of gateway course material. Getting faculty involved in the delivery of an early 
Cl course develops faculty in these areas, and carries on to additional opportunities for further student/faculty 
interactions as students progress through college. 

Derek Bok states that “Many investigators have found that critical thinking and learning in general can be 
enhanced by giving students problems and having them teach each other by working together in groups. 
Where these conditions exist, the great majority of studies show that students make much greater gains over 
those achieved by classmates studying individually or competing with one another.” USCA student focus 
group data, collected in spring 2010 by members of the QEP committee, revealed that students found classes 
that involved discussion, hands on learning, or problem solving to be most memorable. Several studies 
demonstrate the positive impact this type of course instruction can have, not only on first-year students, but 
on faculty efficacy. USCA instructors have repeatedly reported being able to transfer active and engaging 
teaching strategies from a first-year seminar to other courses. This “spill over,” applying Cl teaching 
strategies in an active learning environment, to other courses is one of the desired program outcomes. 

2.1 Survey Results 

We supplemented our Cl pedagogy research by doing our own survey of Cl practices at peer institutions. We 
had strong connections with deans of southeast U.S. business schools, and decided to avail ourselves of that 
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resource, surveying business schools rather than a broader institutional approach. As general education is 
typically a very significant portion of business school curricula, we felt our results would apply to our larger 
range of interest. We conducted an online survey, from the USCA School of Business Dean to Business 
School Deans of 80 institutions, which our Office of Institutional Effectiveness felt constituted a 
representative comparison group. Although we wanted more information, we held the survey to 25 objective 
and one open-ended question. We received 35 responses, 44%, which seemed a favorable response rate. 

When we first began to consider the incorporation of a Cl component to the curriculum, part of the 
underlying impetus was widespread view among faculty that freshmen students were not adequately prepared 
for their transition to the academic challenges ahead. To test this assumption among our peer institutions, 
respondents were asked to indicate their level of agreement with the statement, “freshmen are not adequately 
prepared for success in their courses.” The results are shown in Table 1. 

Table 1. Freshman Preparedness 

Survey Question : “Often educators hear that freshmen are not adequately 

prepared for success in their courses. In your opinion, does this statement 
apply to your entering first-year students?” 


Answer Options 

Responses 

Response % 

Strongly Disagree 

0 

0% 

Disagree 

11 

31% 

Agree 

16 

46% 

Strongly Agree 

8 

23% 

Total Responses: 35 < < > > Skipped Question: 0 


Almost 70% of respondents agreed or strongly agreed with the statement, lending support to the notion 
that freshmen students require additional training to enhance their preparedness for college. Beginning in fall 
2011 we started a pilot Cl program, a one-credit hour course, focused on themes and ideas presented in the 
book The Last Town on Earth, (Thomas Mullen) our freshman first-year reading. Further support for our 
approach is shown on the results in Table 2. Here almost 80% of respondents agreed or strongly agreed that 
a one-credit hour critical thinking methods course would be useful in helping students achieve better 
curricular success. 


Table 2. Value of a One Credit Hour Critical Inquiry Course 

Survey Question : “Do you think a short, one credit hour critical inquiry 

methods course for the first-year students would help them achieve better 
curricular success?” 


Answer Options 

Responses 

Response % 

Strongly Disagree 

2 

6% 

Disagree 

5 

15% 

Agree 

20 

58% 

Strongly Agree 

7 

21% 

Total Responses: 34 < < > > Skipped Question: 1 


Although the respondents indicated support for the inclusion of a one -credit hour freshman level Cl 
course, this method alone was not viewed as the single best approach to help students effectively learn 
critical thinking skills. Instead, as shown in Table 3, respondents indicated the greatest amount of support for 
a program that would incorporate critical thinking skills within the curriculum as a whole. For example, one 
respondent likened the approach of teaching critical thinking to that of teaching writing and speaking (across 
the curriculum), but acknowledged that this could make assessing student learning in this area more 
challenging. Another respondent indicated that faculty on his/her campus wanted to embed critical thinking 
into existing courses, but he/she wondered how faculty could integrate these efforts in a way that would 
result in students developing these skills over time. 


157 





ISBN: 978-989-8533-12-8 © 2012 IADIS 


Table 3. Critical Thinking Skill Development Alternatives 


Survey Question : “In an ideal situation, how would students best learn 

critical thinking skills?” Rank the following alternative options. 


Answer Options 

Best 

Better than 
acceptable 

Acceptable 

Less than 
acceptable 

Worst 

Specifically designed CT course 

8 

3 

11 

10 

2 

Specific areas within curriculum, 
all four years 

15 

12 

4 

2 

0 

Specific areas integrated within 
the major 

3 

14 

14 

2 

0 

Specific areas integrated within 
general education 

4 

8 

10 

9 

2 

Through extra-curricular activities 

1 

0 

6 

6 

19 


Respondents appeared to be supportive of a program designed to integrate critical thinking skills within 
the curriculum, throughout all undergraduate four years, but in practice this did not appear to be taking place. 
As shown in Table 4, only 20% of respondents indicated that critical thinking skills are actually integrated 
within the curriculum for all four years in their respective institutions. Rather, over 42% of respondents 
indicated that although their institutions do not have any specific Cl segments in their curricula, the 
development of critical thinking skills is inherently part of many courses. 

Table 4. Critical Inquiry Curricular Approaches in Use 


Survey Question : “Which of the following does your institution do to 

develop critical thinking skills in your students?” (If more than one fits, 
choose the most influential.) 


Answer Options 

Responses 

Response % 

Through a specific critical inquiry course 

3 

9% 

Specific segments integrated within the 
curriculum, all four years 

7 

20% 

Specific segments integrated within the 
major 

2 

6% 

Specific segments integrated within 
general education 

3 

9% 

No specific program segments, but 
inherently part of many courses 

15 

42% 

Through extra-curricular programs, 
projects and other activities 

0 

0% 

Other 

1 

3% 

No specific critical inquiry program 

4 

11% 

Total Responses: 35 < < > > Skipped Question: 0 


Part of this reluctance to specifically designate any single course as a critical thinking course may be 
based on the perceived difficulty of actually measuring Cl related outcomes and relying on these measures to 
evaluate student performance. As shown in Table 5, over 45% of respondents do not fully believe that critical 
thinking can be objectively assessed, or indicate skepticism about the ability of institutions to assess critical 
thinking with tr aditional standardized multiple choice tests. 
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Table 5. Critical Thinking Assessment 

Survey Question : “Do you believe critical thinking can be objectively assessed? 


Answer Options 

Responses 

Response % 

No 

3 

9% 

Unsure 

5 

14% 

Yes, with standardized 
multiple choice tests 

8 

23% 

Yes, somewhat agree 

9 

26% 

Yes, strongly agree 

10 

28% 

Total Responses: 35 <<>> Skipped Question: 0 


Nevertheless, an intrepid few have actually attempted to surmount the perceived hurdles and offer 
students at least one critical thinking course. As shown in Figure 1, nine respondents (just over 25%) indicate 
that their school or institution had a dedicated critical thinking course in their curriculum. 



Figure 1. Presence of Critical Thinking Course 

While the focus of any Cl course should be on critical inquiry skill development, as we have argued, an 
important factor in the success of the program was tied to faculty development. Consistent with the 
prevailing view on the preferred method of Cl instruction (see Table 3), our goal is that critical inquiry skills 
will ultimately be integrated and assimilated throughout all four years of university study. This goal will be 
achieved in three ways: (1) as a natural outcome of the freshman Cl course. If successful. Cl skills will be 
carried through, by the students, into other classes (and, indeed, their lives); (2) specific other courses, across 
all majors and years, will be developed by interested faculty that specifically focus on furthering Cl skills; 
and (3) full faculty participation in teaching the freshman Cl course. A natural outcome of their training and 
development will result in furthering this process in courses they regularly teach. It is both interesting, and 
appealing, to note that two of these three results are achieved somewhat indirectly. That is to say that as a 
result of planned Cl implementation, both students and faculty will be trained in the process (items 1 & 3), 
with expected and desired extension, ideally, to all other classes. 

The importance of faculty development is illustrated in Table 6. Although sample size for this question is 
limited to only eight respondents, only one of those respondents strongly believed that faculty in their 
institution who teach the critical thinking course receive adequate training and support. For the other 
respondents, the subject of faculty development was noted as the “missing link.” One respondent discussed 
how the lack of faculty development on his/her campus has resulted in professors “conveying their thoughts 
not developing thoughtful students.” Further, he/she said, “Only a serious effort will change that model.” 
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Table 6. Faculty Training & Support 

Survey Question : “In your opinion, do faculty in your institution who teach 
a critical thinking course receive adequate training and support?” 


Answer Options 

Responses 

Response % 

Strongly Disagree 

1 

12.5 % 

Disagree 

2 

25% 

Agree 

4 

50% 

Strongly Agree 

1 

12.5 % 

Total Responses: 8 <<>> Skipped Question: 27 (n/a) 


2.2 Faculty Development 

At USCA, faculty development plans centered on an intensive Critical Inquiry Workshop and a Faculty 
Learning Community. After the workshop experience, faculty would be more knowledgeable about the 
concepts and learning theory underlying critical inquiry instruction. The Faculty Learning Community would 
be continuously available for consultations, and have planned full membership meetings to discuss successes 
and failures, and plans for improvement. This would also provide faculty with access to teaching materials 
and practical applications for developing critical thinking skills in students, which would be accumulated and 
summarized within the learning community. Specific “Faculty Learning Outcomes” follow. 

Faculty Learning Outcomes -> During the workshop, the instructor will: 

1 . Define Critical Inquiry, 

2. Create a sample course outline/timeline for leading the class, 

3. Gain access to a variety of course materials (syllabi, assignments, grading rubrics, technology), 

4. Develop strategies for including Cl in other discipline-specific courses, 

5. Create a plan for incorporating Peer Mentors as an integral part of the instructional strategy. 

After the workshop, the instructor will be able to : 

1 . List the steps necessary for practicing critical inquiry, 

2. Use a variety of activities for fostering critical thinking and collaborative learning in students 
(e.g., discipline-specific strategies, group activities, and active-learning techniques), 

3. Evaluate the effectiveness of the activities for fostering critical inquiry, 

4. Integrate the First Year Reading Experience (FYRE) into the course structure, 

5. Integrate information literacy into the course structure, 

6. Encourage students to appreciate multiple perspectives. 

Incentives to attract faculty to teach Cl courses are varied, with economic incentive seemingly an 
attractive option. In our survey, however, we did not find that direct payment was a widespread approach. 
Only one institution of the eight respondents provided additional pay. The seven others factored the Cl 
course into the regular faculty teaching load. Other compensation options had zero responses. We decided 
on partial course relief. Faculty teach two one hour sections, and receive credit for three hours of instruction. 
We also paid for faculty attendance at a development seminar ($750), but not for teaching the course. We 
also gave Cl faculty priority scheduling for their Cl course, and, importantly, priority by their unit heads in 
scheduling their non-CI courses. We considered offering faculty the option to teach their two one -hour Cl 
courses concurrently, either at the beginning of the term or at the end of the term, giving them half a semester 
of additional non-course time. Ultimately we decided the Cl courses should run the entire semester, but we 
are leaving that option open for future consideration. 

Faculty workshops targeted the skills to help faculty members deliver course elements to achieve our 
desired Student Learning Outcomes (SLO). The workshops provided [suggested] daily plans, resources, 
activities, and teaching methods. Syllabus construction was addressed, including rubric options tied to the 
SLO’s. Additional topics included using technology and social networking as part of the Cl experience. 
Regardless of the specific Cl course direction, a constant emphasis was placed on how to integrate critical 
inquiry into subsequent non-CI specific classes. One way to help promote the integration of critical inquiry 
in the post-CI courses was to plan further workshops on group methods and group project design. 
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Another important element in creating and implementing a successful Cl initiative is broad faculty 
involvement. To that end, every effort was made to involve the entire USCA faculty in the Cl courses. 
Ideally, all full-time faculty, all disciplines, would eventually teach a Cl course. Unit heads were encouraged 
to promote the program and participation, with regular Cl faculty rotations. As an added incentive, we 
decided to avoid detailed course prescriptives, giving faculty the freedom to utilize creative and singular 
approaches in their Cl courses, using their own, individualized teaching philosophies and styles. 

2.3 Pilot Program Implementation 

We held a three-day Cl workshop for an initial cohort of 34 Cl faculty. A critical thinking and teaching 
strategies expert led two days of activities and theory development, followed by a day of course material 
preparation. Later a common course model was integrated into a sample syllabus and distributed to the 
faculty. During the fall 2011 pilot the instructors for the Cl course met three times as a group to discuss and 
review their experiences and opinions concerning the course. At the end of the semester, faculty discussions 
focused on positive aspects of the course, areas in need of review, change, or elimination. The faculty was 
uniformly positive about the design and objectives of the course. Many felt they would like additional 
experience with the delivery of certain critical thinking techniques, and stated they would incorporate some 
critical inquiry techniques into their other university courses. To that end, mission accomplished! 

Some teaching techniques worked very well. Activities such as two-sided debates, role playing, and 
exercises where students are faced with moral decision-making were particularly popular. All of these 
activities were more likely to be successful when students worked in small groups that had to reach 
consensus on decisions. The faculty using student peer mentors as assistants in the course were enthusiastic 
about the peer mentor contributions. Faculty also identified a number of teaching techniques that should be 
discouraged because they prevent collaborative effort and prevent active participation by the entire class. 
Among those techniques were traditional lectures and open-ended questions addressed to the entire class. A 
sampling of 165 student reflective essays (a requirement of the course) revealed strong support for the class 
overall. A large number of students felt the critical thinking exercises helped them gain greater appreciation 
for different perspectives. A common comment was appreciation for the library literacy exercise because it 
opened their eyes to new ways to conduct class research. They also commented that the in-depth discussions 
on topics from the First Year Reading book were novel and enjoyable experiences and provided them with 
new ways to consider the reading of a book. Overall, it appears that the content of the course was well 
received by the students. 

Our critical inquiry class highlighted student questioning, data gathering, and analytical evaluation. Many 
of our students had never encountered a learning environment that asked for and embraced their opinion. 
Ground rules for the class were clear: everyone has an opinion; an opinion grounded in research; all opinions 
are respected. Creative approached were encouraged. For example the first class assignments was for groups 
to interpret photographs, such as a man standing with a black case. The initial attempt at this exercise resulted 
in comments such as “it’s just a guy waiting for a bus”. After each group gave their response, they were 
asked to change groups and then re-evaluate the pictures. The next evaluations resulted in comments such as 
“he is a spy and there are secret documents in the case”. The class then discussed the ideas of objectivity and 
subjectivity and finished with a review of validity and truth. These are serious concepts for individuals that 
have never been exposed to this type of inquiry. The class was also exceptional in that it gave a cohort group 
the chance to explore together. Most of the class time was spent on discussion and lively debate, which arose 
spontaneously from class room discussion. It took two or three weeks for the students to feel comfortable and 
to trust themselves, fellow students and instructor with this new process. A key facet of the course was to 
teach them that they are “allowed” to be creative and search for new and different ideas. 

Faculty learning community sessions yielded valuable feedback. Instructors were positive about the 
course, but recognized that critical thinking improvement were not likely to be observed in the short term. 
The course was considered valuable in opening new learning activities and pedagogical styles for the 
instructors. Most instructors commented that they would use the same techniques in other courses. For 
example, one instructor learned more of the merits of course structure vs. open dialogue. On reflection he 
felt his typical content-laden lectures were less productive than he thought. He planned for more of the 
activity-based, group learning, with high outcome accountability for future classes. Others agreed and felt 
this type of learning was well received by the students, with the added value of better student/teacher 
interactions. 
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3. CONCLUSION 

Survey results indicated that there is widespread concern about the level of preparation exhibited by 
incoming freshmen. Although most respondents believed that the integration of Cl throughout all four years 
of undergraduate education was optimal for Cl development, there is also strong support for the incorporation 
of a dedicated Cl course in the first year curriculum. Most institutions surveyed did not have such a course. 
Part of the reluctance to offer such a course may be related to the perceived difficulty in assessing critical 
thinking skills using traditional classroom methods. There also appears to be inadequate training and support 
for faculty assigned to Cl courses. The proposed Cl faculty development program described for USCA was 
designed to alleviate some of these obstacles. Only through implementation and ongoing assessment can we 
judge the quality and effectiveness of the Cl program described. Our critical inquiry course will hopefully 
set in motion a lifelong philosophy of student discovery and analysis. The philosophy must be incorporated 
in the entirety of the educational process, with each new critical inquiry experience adding to the last. 

After the pilot year of the Cl program, instructor morale was very high and student feedback was positive. 
Instructors were able to experiment with flexible teaching methodologies and approaches, and most 
commented that their experiences would positively affect the other courses they teach. Student feedback 
indicated their appreciation of the focus on critical thinking and the introduction to new learning skills. 
Student also tended to favor the same teaching methodologies favored by the instructors. The effectiveness 
of the Cl course at USCA can best be judged when each cohort of Cl students is assessed at graduation. We 
will be use freshmen-senior assessment (ETS-PP testing) to those ends. Quantifiable answers, however, may 
remain elusive, as isolating the singular effects of a freshman Cl course four years later is surely confounded 
by multiple factors. We firmly believe, however, that the introduction to critical inquiry as freshmen will 
have lasting effects, especially on under-prepared students. By introducing freshmen to critical inquiry and 
sound exploratory learning practices, and by interjecting critical inquiry methods throughout the curriculum, 
we believe senior students will be more intellectually capable and inquisitive at graduation. We remain 
convinced that the development of a critical inquiry program for freshmen is in the best interests of the better 
undergraduate experience we all seek. 
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ABSTRACT 

This paper presents the requirements work carried out as part of developing an intervention to improve students’ critical 
thinking skills using location-based mobile learning. The research emerged from seeking to identify ways of getting 
Interaction Design students into real world environments, similar to those in which they will eventually be designing, 
maximising their ability to identify opportunities for innovation. 

The first stage in designing the system is to conduct a comprehensive requirements study to understand specific student 
and staff needs in the envisaged scenario. As part of the requirements study we were interested in identifying weaknesses 
in the current mode of teaching, as well as problems students currently experience in understanding key concepts. We 
found that students did not stay adequately focused on the task and failed to identify appropriate technologies. We also 
noted scope for further support with analysing their findings. This information is helping us to determine the type and 
nature of location-based hints and formative feedback that the system can provide to aid students’ understanding of the 
context they are in. 


KEYWORDS 

Location-based, mobile learning, critical thinking, requirements analysis 


1. INTRODUCTION 

Since the early days of mobile learning research, there has been a wide debate on its exact definition. Various 
researchers have different interpretations. Many are eager to show that m-learning is not a reduced version of 
e-learning (Belshaw, 2011). Vavoula et al. (2004) have defined it as “Any sort of learning that happens when 
the learner is not at a fixed, predetermined location, or learning that happens when the learner takes 
advantage of learning opportunities offered by mobile technologies”. Furthermore, Woodill (2011) 
acknowledges that there is a shift in the perception of mobile learning, “Ten years ago, mobile learning was 
about displaying e-learning on a small screen”. He argues that it allows learners to learn in an “anywhere, 
anytime” manner and to access information when needed. 

The strength of mobile learning lies in taking advantage of the rapidly evolving scope of mobile 
technologies. Being able to sense the context and location of the learner has opened up a wide range of 
possibilities for researchers to create more engaging, contextualised and personalised learning activities, 
maximising the benefit of the learning experience. Personalisation is one of mobile learning’s strengths. 
According to Kinshuk et al. (2009), personalisation could be acquired either by adapting to the learner’s 
characteristics, learning styles, performance, and needs; or by adapting to the context in which the learning is 
taking place. 

This paper presents the requirements work carried out as part of developing an intervention to improve 
students’ critical thinking skills using location-based mobile learning. The idea for the research emerged 
from seeking to identify ways of getting Interaction Design students into real world environments, similar to 
those in which they will eventually be designing, in order to enhance their ability to identify opportunities for 
innovation. However, sending students out into real-world environments with a brief to be evaluative and 
analytical, without the presence of a teacher, can lead to a superficial and frustrating experience, especially 
for students with beginning levels of analysis and limited critical thinking skills. It is not always possible for 
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teachers to accompany students, and moreover, rather than immediate input from teachers, prompts to 
provoke the development of their own thinking might be more beneficial. 

In order to design the system, the first stage is to conduct a comprehensive requirements study to 
understand specific student and staff needs in the envisaged scenario. As part of this study we were interested 
in identifying weaknesses in the current mode of teaching and problems students experience in understanding 
key concepts. This information is helping us to determine the type and nature of location-based hints and 
formative feedback that the system can provide to aid students' understanding of the context they are in. It is 
important to ensure that students don't miss out on key areas that may help them with analysing the situation 
properly. The hints can also give them the beginning of the threads to developing innovative ideas, thus 
providing added value to the mobile-based learning system. 

The next section of the paper draws on the literature to explain context-aware and location-based 
learning, and the relevance of situated learning and critical thinking to this study, summarising the related 
work in the area. This is followed by an outline of the research methodology detailing the requirements 
gathering process and insights gained and explaining how these have been incorporated into the initial 
prototype designs of the application. 


2. LITERATURE REVIEW 

2.1 Context- A ware and Location-Based Mobile Learning 

Context-aware computing is a rapidly growing research area. It aims to promote a flowing interaction 
between humans and technology (Barkhuus and Dey, 2003), collecting information from the surroundings of 
the user to provide an understanding of what is currently happening (Naismith et al., 2004). 

Abowd et al. (1999) defines a context-aware system as follows: “A system is context-aware if it uses 
context to provide relevant information and/or services to the user, where relevancy depends on the user’s 
task”. Context, according to Brown et al.’s (2010), is “the formal or informal setting in which a situation 
occurs; it can include many aspects or dimensions, such as location, time (year/month/day), personal and 
social activity, resources, and goals and task structures of groups and individuals”. 

Barkhuus and Dey (2003) define three levels of context-aware applications depending on the interactivity 
with the user: 

1. Personalisation: the user determines the way the application behaves in a particular situation. 

2. Active context-aware: the application changes the content independently, based on the sensor data. 

3. Passive context-aware: the application presents the changed context, sensor data, to the user and 
lets him/her take control of decisions about how the application behaves. 

Much research has shown the significance of context-awareness in education (Yau and Joy, 2009; Fisher 
et al., 2009; Ghiani et al., 2009; Bhaskar and Govindarajulu, 2010; Chiou et al., 2010). Fisher et al. (2009) 
argue that the use of mobile devices such as tablet PCs in education can enhance the teaching experience of 
lecturers, as well as the quality of the learning experience of students. Shih et al.’s (2010) research indicates 
that using mobile learning helped lower the cognitive load of students with low achievement rates. 

A large area of context-aware mobile learning research has been focussing on museums and tours in 
providing information based on the person's location (Reynolds et al., 2010; Chiou et al., 2010; Yatani et al., 
2004; Costabile et al., 2008; Park et al., 2007; Hsu and Liao, 2011). According to Reynolds et al. (2010) 
many students appreciated the contextual information offered by the mobile device which encouraged them 
to ask more questions. This enhanced their knowledge about the objects in the museum. 

From a pedagogical perspective, context-aware and location-based mobile learning is clearly related to 
the situated learning theory: it is important to gain an understanding of this in order to learn how to optimise 
the development and implementation of context-aware and location-based applications. 

2.2 Situated Learning 

Lave and Wenger’s (1991) situated leaning paradigm, states that the situation in which learning occurs has a 
great effect on learners. They argue that learning must not be abstract and out of context. It is situated, and 
takes place within the context, activity, and culture in which it occurs as a “legitimate peripheral 
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participation” process. Lave and Wenger (1991) emphasise social communication and interaction as being 
significant parts of situated learning. Learning should be presented in an authentic setting supporting 
knowledge exchange between learners (Naismith et al., 2004). 

Defining the key characteristics of situated learning can differ between disciplines and technologies 
(Yusoff et al., 2010). When designing situated learning using mixed reality technology, Yusoff et al. (2010) 
outline three main elements: authentic context, authentic activity/task, and users’ collaboration. Lunce 
(2006), in designing situated learning using simulation, suggests four criteria for situated learning: a specific 
context that impacts learning must be defined; peer-based interactions and collaboration between students 
must take place; knowledge is tacit; and tools must be used to accomplish real-time objectives. 

Herrington et al. (2000) propose the following elements for situated online -learning using multimedia: 
“authentic contexts and activities, access to expert performances and the modelling of processes, multiple 
roles and perspectives, collaborative construction of knowledge, coaching and scaffolding, reflection to 
enable abstractions to be formed, articulation to enable tacit knowledge to be made explicit, and integrated 
authentic assessment”. 

In summary there is agreement that although technologies differ, for a successful learning experience, 
situated learning has to take place in an authentic setting, with authentic contexts and activities. Therefore it 
is vital that this research is aligned to, and integrated into real teaching and learning scenarios to ensure 
validity. Additionally, facilitating collaboration between learners can be an important enhancement of the 
learning experience. 

2.3 Critical Thinking 

As this research proposes to encourage and develop students' critical thinking and analysis, it is important to 
define what this means. There are several relevant definitions of critical thinking, some as early as Dewey 
(1933). However, for the purposes of this study one definition has been identified, that of Scriven and Paul 
(1987), who defined it as "the intellectually disciplined process of actively and skillfully conceptualizing, 
applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, 
experience, reflection, reasoning, or communication, as a guide to belief and action". Their definition shows 
a clear relation to Bloom’s taxonomy, as it relates critical thinking to the three higher levels of the taxonomy 
(analysis, synthesis, and evaluation) (Duron et al., 2006). This definition emphasizes the multifaceted nature 
of critical thinking, expressed through a number of activities. These activities correspond to the assessed 
work carried by students in this study explained in section 3.1 


3. THE SITUATED LEARNING PROBLEM DOMAIN 

This intervention aims to resolve issues faced by students when learning in a real-world situation. The initial 
situated learning activity is being developed for a level 2 Human-Computer Interaction (HCI) module in the 
Department of Computer Science and Creative Technologies at the University of the West of England 
(UWE), Bristol. As part of their work for this module, students are required to evaluate and carry out a 
context-based analysis as part of a requirements gathering process for a computer-based system. The 
requirements gathering process is to explore opportunities for a technological intervention, and also to ensure 
that the solution developed will suit the particular situation/users; so the emphasis is on gaining a really deep 
understanding of this: the people involved, their activities and the context. The student designer needs to 
consider the question: ‘what are the constraints and barriers within the situation that need to be addressed?’ 

To facilitate this, the PACT (People, Activities, Context, and Technology) framework is used to prompt 
students to consider specific categories in their analysis. The elements of the framework are described by 
Benyon et al. (2010): 

1. People: they differ physically, psychologically, and in terms of their knowledge of 
technology. 

2. Activities: they differ in terms of temporal aspects (response time, frequency of the 
activity, time pressure and peaks), cooperation, complexity, and safety-criticality. 

3. Contexts: the different environments in which the activities take place encompass the 
organisational and social context and the physical environment. 
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4. Technologies: these should reflect the specific issues identified in considering the previous 
elements. Features include input, output, communication, and content. 

The students’ brief is to go into specific environments relevant to their design task and to collect data 
regarding the first three elements of the PACT framework, using mainly observation, but also formal and 
informal interviews, questionnaires and focus groups. They then need to analyse critically the data collated in 
order to identify possible constraints and solutions. 

As stated earlier, this research is seeking to investigate a mobile location-based system to support the 
students’ activity. The next section details our findings from the requirements gathering work carried out. 

3.1 Research Methodology 

Requirement gathering is a significant part of any user-centred design (Lazar et ah, 2010); it aims to establish 
a deep understanding of the situation, to refine the user requirements and identify the functional and 
non-functional requirements of an application. In order to improve validity, a range of approaches was used, 
enabling us to triangulate the findings as part of the analysis. The approaches include the analysis of the 
students’ submitted assignments, observations of teaching and interviews with the lecturers. Figure 1 
illustrates this approach. 



Figure 1 . Requirements approach 

3.1.1 Analysing Students’ Submitted Assignments 

Undergraduate students in their second year of the Web Design course in the Department of Computer 
Science and Creative Technologies at the University of West of England were required to submit a portfolio 
of small assignments. 47 out of 48 students submitted the part of their portfolio considered here. The work of 
these students was anonymised prior to analyses. Each student’s work was separately scrutinised to identify 
his/her weaknesses and any good practice. It is crucial to know how common a particular issue is among the 
students to gain an understanding of whether that issue needs to be considered when designing and 
developing the application. The analysis was verified by checking its correspondence with written feedback 
from the lectures on each aspect of their work. 

To anonymise the students, each was given a number from 1 to 47. The table below shows the issues 
identified and occurrences, identified by the student number. 

Table 1. Issues and occurrences 


Issue 

ID 

Issues 

Occurrences 

Total number 
of students 

A 

No clear links of the issues discussed in P, A, and C to 
Technologies. 

2,3,4,7,8,11,12,14,17,20,2 

2,24,26,27,29,34,45 

17 

B 

Some issues were not related to the right element of PACT 

2,3,4,7,8,9,11.13,17,26,47 

11 

C 

No real consideration of the human factors. 

3,4,10,13,22,23,28,29,35, 

44 

10 

D 

Issues were general and not mainly context related 

2,4,13,22,23,30 

6 

E 

No links of the issues discussed in P, A, and C with 
Technologies. 

4,10,23,32,33,36,41,44 

8 
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F 

Gave the issue with the solution rather than putting the solution 
under Technologies and linking it to P,A or C. 

11,17,24,26,27,37,41,45,4 

6,47 

10 

G 

Need more thoughtful consideration of the context 

12,29,32,41,44, 

5 

H 

Need to address issues found under each PACT element. 

13,28,29 

3 

I 

Technology issues could be expanded 

16,17,35 

3 

J 

Linking should be more explicit 

18,19,21,30,46,47 

6 

K 

Need to find solutions to current problems not eliminate ideas 
because of that problem. 

28,19 

2 

L 

Need to think about and address issue from observations, 
experience. 

28 

1 

M 

Need to identify issues under each P, A, C, and then see what T 
can allow for the proposed self-checkout not for the cafeteria. 

31,32,34,35 

4 

N 

Need to consider human factors in more depth. 

33,34,35,41 

4 


In the table above we can see that 36% of the students had difficulties linking the characteristics of the 
people, activities, and context identified to technologies. In other words they should have identified the 
technologies that would serve the characteristics of the people carrying certain activities in that particular 
context. Moreover, 25% of them had issues with understanding the PACT framework itself. However, it 
should be noted that the lecturers had not put a great deal of emphasis on this, as mentioned below in the 
section. Observations of Teaching. It is clearly important to consider the people who will be using the 
technology; nevertheless, 21% of students did not give this much attention. The chart below shows each issue 
with the corresponding percentage of students to whom this applies. 



Issue ID 


Figure 2. Percentages of the occurrences of issues 

As future work we hope to organise a focus group to elicit further information about the difficulties 
encountered by students from a pragmatic point of view, and the functionalities that they would wish to see 
in the application to help them overcome these difficulties. We also hope to cross-reference this information 
with the information obtained from the students’ coursework analysis presented above. 

3.1.2 Interviews with Lecturers 

Our aim was to explore the following issues with the lecturers teaching the HCI module: 

1 . Their current practice of teaching students, especially concerning the PACT framework' 

2. Their current approach to explaining assignment to students, 

3. The students’ current practice in completing the assignment, the difficulties they encounter, and the 
reasons behind these difficulties from the lecturers’ point of view, and 

4. What they hope this intervention will achieve. 

A series of unstructured interviews with two lecturers teaching the HCI module was carried out. This is a 
significant part in the requirements gathering as it highlights clearly the functionality of the application that 
needs to be considered. 

Current practice of teaching 

In the Human-Computer Interaction module, students learn about how people undertake activities in 
context using technologies. They apply the PACT (People, Activities, Context, and Technology) framework 
to analyse situations in order for them to design interactive systems. The lecturers explain the PACT 
framework in detail to students giving them specific examples to clarify the concept. These include scenarios 
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such as (a) a female student using her smart phone to send a text message whilst on a moving bus, when she 
is seated, when she is standing holding on to a bag and an overhead strap for balance, and when the bus is 
extremely crowded, and (b) and elderly woman setting her burglar alarm which is located in a dimly lit 
passageway, with situations where the elderly woman has different age-related conditions. These example 
scenarios are formulated to support the students in understanding the elements of the framework. 
Photographs are shown to provide students with a realistic view of the physical environment. However, the 
weakness is that the students are not able to immerse themselves in the actual environment to get a tangible 
understanding of the constraints, and therefore fail to develop empathy for the users. 

Explaining the assignment to students 

As part of one of their assignment activities, students are required to conduct a requirements study for the 
design of a new technology. In the past this has included the design of a university information kiosk and a 
digital guide for a music festival. This year students were asked to consider the design of a self-service 
checkout for use in a cafeteria. Using the ‘PACT’ framework, students were required to analyse the factors 
that they would need to take into account in designing such a system. They were required to gather data for 
the analysis via observations of the OneZone cafeteria (Main University Cafeteria at UWE) at various times, 
consider their own experiences, as well as conduct short interviews with at least 3 stakeholders. 

Students were then required to present their findings as a mind-map/spider diagram, ensuring that there 
were clear links between the People, Activities and Context elements and the Technologies considered. They 
needed to explain in separate paragraphs and in relation to each element of the PACT framework, why the 
points that they had noted were of significance. 

This was explained to the students in class and described on their coursework assignment specification 
alongside the marking criteria. 

The students’ current practice of the assignment from the lecturers’ point of view 

It is crucial to investigate the lecturers' understanding of the students’ current practice of the assignment, 
the difficulties they encounter, and the reasons behind it from their point of view. Lecturers assume that 
students get distracted by the environment and sometimes forget the main purpose of their assignment. From 
their experience, students miss out key details when carrying out their analysis, leading to a disconnected 
analysis, especially between the elements of PACT. Moreover, students tend to forget that ‘people undertake 
activities in context using technology'; as a result they fail to consider the implications of what they have 
identified for each of the elements. People, Activities, and Context, in relation to the Technology. They thus 
miss the purpose of their assignment, to analyse the situation and consider technologies that reflects peoples’ 
needs when carrying out certain activities in a particular context. In some cases, students fail fully to engage 
with, or appreciate the relevance of going to the location at all, and complete the activity in a rushed manner 
with little or no reflection. 

The lecturers’ view of the intervention 

The lecturers want this mobile application to assist students when carrying out their analysis. They want it 
to provide students with prompts when they are at the location. These prompts should address the students’ 
weaknesses already identified by the lecturers and also from the analysis of the previous students’ 
assignments, discussed earlier in the paper. The lecturers suggest that the students should be able to capture 
images using the application, take notes, and track their own progress. 

3.1.3 Observations of Teaching 

In addition to the interviews with lecturers, observation of teaching was conducted. This gave us a better 
understanding of the current practice. Attending HCI lectures was a valuable part of the research, giving an 
insight into how students engage with the lectures and what question they might raise about the PACT 
framework and the assignment. Observing the collaboration forum on Blackboard was also useful, revealing 
students’ queries and concerns and the feedback given by the lecturer. Students expressed concern was about 
the elements of PACT and the relationship between the different elements. This raised significant questions 
that needed resolving. To what extent is it crucial to encourage students to use the PACT elements correctly? 
Is it a tool for bringing to light many factors or do we value it as a categorisation tool? It was important get 
back to the lecturers to discuss those two issues. It was agreed that we should remind students of the PACT 
elements without putting undue emphasis on categorisation. 
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3.2 Findings 

This section will explain the insights gained so far. It also explains how we are translating these into design 
features. 

Students lose focus on the purpose of tasks when away from classroom. They may get distracted by their 
surroundings and miss out key elements. So a key feature of this mobile application could be to assist 
students of the purpose of their learning and support their progression through the activities in a personalised 
manner. 

When students reach a pre-specified location, the application should display a detailed map identifying 
the various sub-locations and containing either text and/or images. These hints could be designed to aid them 
in widening their perspectives, in developing their own ideas and in critical evaluation. The text notes could 
vary from simple instructions and prompts, to questions, and in some cases to links that will open a quiz 
webpage; the particular content would depend on the specific aspect that the lecturer would want the students 
to focus on. 

It is important to encourage students to think of issues beyond their own realm of experiences and 
perspectives. Providing students with functionality to share comments, ideas and perhaps stories if desired, 
may enable them to benefit from their peers’ knowledge and different perspectives. Adding a collaborative 
learning aspect to the activity, students will be able to post their comments for their lecturers and fellow 
students. 

Students have varying levels of ability when it comes to design thinking, and work at different rates. A 
mobile application provides opportunity for personalised learning, such as paced progression, checklists to 
give a sense of achievement and motivation, and structured disclosure, based on the students’ level of 
interaction with the application. 

Some students have been found to struggle in analysing their findings and specifically in using their 
findings to develop new ideas. Prompting them with probing questions that challenge their assumptions or 
get them to explore other methods of requirements gathering, beyond observation could help them identify 
innovative opportunities. This approach could also address the problem of their failing to identify appropriate 
technologies for the specific characteristics that they have identified earlier. 


4. CONCLUSION AND FUTURE WORK 

Research in mobile situated learning in higher education is still in its beginnings. Our research has potential 
to add to understanding of how mobile applications can assist students learning in-situ. This paper outlines 
the background to the research, discussing some key concepts. It focuses on the data obtained from the 
interviews conducted with the lecturers of the HCI module, the observations of teaching, and analysis of the 
students’ submitted assignments; this data has confirmed the difficulties that students encounter whilst 
carrying out their coursework, helping us to establish the functional and non-functional requirements to be 
considered when designing and developing the mobile application. We are continuing with the research, 
adopting a user-centred, iterative approach to the design. We are currently working on the first prototypes of 
our application and will evaluate these, applying such usability criteria , as how easy the features of the 
application are to understand, its learnability, the effectiveness of feedback and the ease of interaction. 
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ABSTRACT 

This paper reports on the development of a new information communications technology (ICT) learning preference 
survey, its cross-validation with attitudes towards mobile learning, and new perspectives on information seeking, 
information sharing, and mobile access derived from the relationships uncovered. The Information and 
Communications Technology Learning (ICTL) survey instrument is designed to measure learners’ preferences and 
information behavior for learning, knowledge acquisition, and sharing in the Web 2.0 technology pervasive 
environments of the twenty-first century. The instrument development process includes analysis for reliability and 
validity of instrument scales. The multi-step refinement process revealed two constructs with, respectively, very good 
and respectable measurement properties: Information Sharing (alpha = .83) and Information Seeking (alpha = .71). 
Cross-validation with a newly established mobile learning instrument confirms that both the Information Sharing and 
Information Seeking scales of ICTL have significant (p < .001) alignment with mobile learning attitudes. Implications 
of findings are discussed. 


KEYWORDS 

ICT learning, information seeking behavior, mobile learning. 


1. INTRODUCTION 

Collaboration fostered by a doctoral level psychometrics course and scholarly interest in the role of mobile 
technologies in learning prompted the authors to develop and validate an instrument to measure learners’ 
dispositions toward new learning and knowledge acquisition activities that are made available by information 
and communications technology (ICT). The authors also sought to examine relationships between the ICT 
survey and an established mobile learning survey. 

The two surveys employed in this research study were: 

1) The Information and Communications Technology Learning (ICTL) survey, and 

2) The Mobile Learning Scale (MLS). 

The development of the Information and Communications Technology Learning survey is described in 
detail in subsequent sections of this paper. The Mobile Learning Scale vl.O also plays a key role in this study 
and therefore its development history and psychometric properties are briefly discussed. MLS vl.O was 
originally created from the key points developed for a paper on mobile learning prospects for informal 
learning in higher education (Khaddage & Knezek, 2011). Many of these key points initially emerged during 
working group discussions of the 2011 International Summit on ICT in Education (UNESCO, Paris, 2011) 
(Knezek, Lai, Khaddage, & Baker, 2011). Explorations of the relationships among the emerging factors 
uncovered the relationships that are reflected in the title of this paper and are also addressed in details in the 
discussion. These findings illustrate that cross-validation techniques can contribute to the richness of 
understanding as gauged by the underlying constructs found in the cross-validation of instruments with 
established reliability. 


171 



ISBN: 978-989-8533-12-8 © 2012 IADIS 


2. CONCEPTUAL RATIONALE 

Measuring student technology attitudes related to learning and information seeking in technology pervasive 
learning environments of the twenty-first century is of particular interest to educators and learning 
technologists who seek to integrate ICT for effective teaching and learning. Also of great interest are 
learners’ attitudes towards learning with mobile technologies. The role of mobile technology in education is 
yet to be clearly defined, and therefore it is reasonable that many current papers present convincing 
arguments about ways that that mobile technologies and applications can bring many benefits to the learning 
environment. For example, Khaddage and Latterman (2012) point out that mobile technologies and 
application offer benefits such as portability, simplicity, and availability. A multi-year study conducted by 
Kolko and colleagues at the University of Washington reports on the adoption of ICT in Central Asia stating 
that “mobile phone usage is outpacing the rate of Internet adoption, that access to the Internet is primarily 
through public access sites”. Also reported is a trend towards the combination and conflation of technology 
related information seeking and communication usage patterns (Kolko, Rose & Johnson, 2007). 

The seemingly unlimited new media contexts and information access options in formal and informal 
settings are often regarded as venues of invaluable opportunity for teaching and learning in the twenty-first 
century (Arnone, Small, Chauncey & McKenna, 2011). There are those who warn that education that does 
not utilize these new media-based technologies will not adequately develop student potential, leaving 
students with a deficit in digital literacies that may contribute to student disengagement from classroom 
activities (Judson, 2010). Felt (2010) identifies new media literacy (NML) skills as a set of cultural 
competencies, social skills, and ICT tool skills that are essential for learning and mastery in the twenty-first 
century. New media literacy skills are being associated with content-based learning in the framework of 
Jenkins and colleagues (Jenkins, Clinton, Purushotma, Robison, Weigel, 2006), and are thought to support 
student thinking and learning. 

With information communications technology (ICT) now commonplace in all facets of our daily lives, 
including teaching and learning (Christensen & Knezek, 2006), it is important to understand student 
information seeking behavior in the Web 2.0 framework for formal and informal learning. Kuhlthau's 
Information Search Process (ISP) model, devised in the 1980’s and revised in the 1990’s, depicts six stages 
of student information activity: initiation, selection, exploration, formulation, collection, and presentation 
(Kuhlthau, 1991). These six stages reflect the primary tasks to be accomplished when students go beyond 
social communications technology exchanges and engage in complex information behaviors that can be 
useful for directing learning activities. Emphasis should be placed on supporting information seeking tasks 
which are associated with independent learning and knowledge construction (George, Bright, Hurlbert, 
Linke, St. Clair & Stein, 2006). Established for an understanding of student information behavior in support 
of guided instructional, diagnostic, and intervention programs, Kuhlthau’s ISP model has been applied 
extensively for decades. An extensive review of literature and inquiry project conducted in 2007 among 
( n=574) school students indicated that Kuhlthau’s information seeking model for affective, cognitive, and 
physical dimensions of information behavior continues to be usefulness for explaining student information 
seeking behavior and knowledge acquisition in the digital, technology pervasive information environment of 
Web 2.0 (Kuhlthau, Heinstrom, & Todd, 2008). Findings confirm that while some stages in the search 
process may be intensified in new instantaneous information environments “the Information Search Process 
seems to be an over-arching process regardless of search venue, or print or digital format” (Kuhlthau, et al, 
2008). 

Mobile learning is a relatively new phenomenon with its theoretical basis still under development 
(Kearney, Schuck, Burden, Aubusson, 2012). With the rapid growth of mobile devices throughout the world 
(World Bank, 2012), the need has emerged for studies of affordances and barriers that might enhance or 
constrain the adoption of mobile learning in higher education. This approach is consistent with the principles 
of diffusion of innovations (Rogers, 2003) that have served as a useful model for studying the introduction of 
new information technologies in the past, and which have been suggested by Chueng and Hew (2009) as an 
appropriate framework for examining the adoption of mobile devices in teaching and learning. 

While mobile learning is recognized as the desired outcome for application of mobile technology to 
distance learning, the widespread availability of mobile technology and the association of mobile learning 
with new forms of communications related activities for engaged learning spotlights mobile learning for a 
broad range of formal and informal learning activities in the twenty-first century (Andrews and Tynan, 
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2012). Cox (2012) contends that research approaches investigating innovative ways of teaching and learning 
with ICT in the future should address technology-enhanced learning “outside formal educational settings” (p. 
2) as well as the opportunities presented by “the uptake of thin client technologies” (p. 5) that are mobile and 
personalized. Lai (2011) devoted a section of his work on digital technology and the culture of teaching and 
learning in higher education to “... how digital technologies may provide a more active and flexible learning 
experience by adopting a participatory pedagogical approach and by blending formal learning with informal 
learning” (p. 1263). 


3. DEVELOPMENT OF THE INFORMATION AND COMMUNICATIONS 
TECHNOLOGY LEARNING (ICTL) SURVEY 

The Information and Communications Technology Learning (ICTL) survey was designed and validated to 
help address questions related to how students prefer to utilize ICT for information seeking, information 
sharing, and knowledge acquisition. Instrument development included analysis for internal consistency 
reliability, principal components exploratory factor analysis, multidimensional scaling, and higher order 
factor analysis. Items were initially gathered by the first author in an effort to better understand learner 
preferences and the role that ICT plays in learning today. A review of literature did not reveal validated 
instruments for measurement of student learning preference for use of ICT for learning activities and 
knowledge acquisition within the ubiquitous communications landscape of the twenty-first century (Mills and 
Knezek, 2012). Items for a 15-item prototype instrument, the Information Communications and Technology 
Learning (ICTL) survey version 1 .0, are displayed in Figure 1 . Survey questions are Likert-type, rated on a 5 
point scale with response choices from 1= strongly disagree to 5 = strongly agree. Note that item 9 is 
negatively worded requiring that response ratings be reversed before summing (or averaging) with ratings 
from other items to form a Likert-type scale. The reverse coding for item 9 results in answer choices having 
increasing value along the response scale 1-5. 


Information and Communications Technology Learning (ICTL) survey 

1 . 1 would like to be a participating member of an online community. 

2. 1 use Internet technology to explore topics of interest. 

3. 1 like to share interests and reflections online. 

4. 1 like to enroll in classes to continue my education. 

5. 1 use Internet communications and other technology tools for self-expression. 

6. 1 learn many things by interacting with other Internet users. 

7. 1 like to take classes from good professors. 

8. 1 use Internet communications technology tools when I want to learn about something new. 

9. 1 learn best in a traditional classroom setting. (R) 

10. Internet technology helps me be successful in my college classes. 

1 1 . More classroom learning should include interactive communication technology experiences. 

12. The things I need to know are taught by instructors in the classroom. 

13.1 leam more when I regulate my own learning experience and seek information on things that I want to 
learn about. 

14. 1 use Internet communications technology to keep current on topics related to my field of expertise. 

15.1 post information that might be of interest to other people. 

Figure 1. Information and Communications Technology Learning (ICTL) survey items. Note: ICTL vl.O 2011 by L. 

Mills & G. Knezek. 


4. DATA COLLECTION 

Survey subjects were volunteer participant college students enrolled in one of two institutions of higher 
education in Texas (USA), a junior college with a two-year curriculum, and a four-year state university. 
Sixty-two (62) respondents completed the ICTL survey online during the fall semester of 2011. The survey 
participants were 89% women (n=55) and 11% men (n=7) spanning 18 to 59 years of age. 
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5. INSTRUMENT REFINEMENT 


5.1 Internal Consistency Reliability and Construct Validity 

Reliability analysis was used to assess the internal consistency of the instrument scales according to 
Cronbach's Alpha index. As shown in Table 1, Cronbach's Alpha for all 15 items of the ICTL instrument is 
considered “respectable”, (alpha = .77) according to reliability guidelines by DeVellis (1991). 

Exploratory factor analysis, principal components analysis (PCA) with varimax rotation, was conducted 
in order to identify scales/factors that are orthogonally aligned (Mertler & Vannatta, 2005). PCA produced 
four factors with eigenvalues greater than one. Scree plot analysis (see Figure 2) indicated a two or possibly 
four factor solution in light of Stevens’ (1992) suggestion that constructs in the sharp decent of the graph, 
before the first point of leveling, be retained. The four-factor solution was selected because the items within 
each factor were judged to have greater content validity. These four factors (see Table 1) were tentatively 
named Online Reflection, Internet Exploration, ICT Research, and Classroom Learning, with alphas of 0.88, 
0.70, 0.43, and 0.54, respectively, when subjected to post hoc internal consistency reliability analysis. 
Measurement scales produced for these factors were found to have internal consistency reliabilities ranging 
from very good for Online Reflection to unacceptable for Classroom Learning, according to guidelines by 
DeVellis (1991). 


Table 1. Cronbach’s Alpha for ICTL. Internal consistency reliabilities for ICTL scales 


Scale 

No. of 

Item 

Cronbach’s Alpha 

DeVellis 


Items 

numbers 


Guidelines 

ICTL- Total scale 

15 

1-15 

0.77 

Respectable 

Online Reflection 

6 

3,5, 11,6, 
15, 1 

0.88 

Very Good 

Internet 

Exploration 

4 

7, 2, 13,4 

0.70 

Respectable 

ICT Research 

3 

8, 10, 14 

0.43 

Unacceptable 

Classroom Learning 

2 

12, 9 

0.54 

Unacceptable 


Scree Plot 



Figure 2. Scree plot for exploratory factor analysis of ICTL items 

Higher order factor analysis was conducted to explore explanatory constructs at a higher level common 
among the four factor scales identified by PCA: Online Reflection, Internet Exploration, ICT Research, 
Classroom Learning (see Table 2). Scale scores produced for each of the factors were used as input variable 
values for the higher order procedure. A Scree plot for higher order factor analysis (Figure 3) reveals two 
factors in the sharp decent of the line. Table two shows that these two components explain 70% of the total 
variance. 
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Table 2. Higher order factor analysis loadings for four ICTL scales. Two components account for 70% of variance 


Total \ a nance Explained 

Extraction Stum of Squared Rotation Sums ofSqnared 



Initial Eigenr alues 

Loadings 

Con- 

poDeU 

1 

Total 

1567 

4» of Cumulative 

Van 9* 

39.182 39.182 

Total 

L567 

2 

1327 

30.686 69.868 

1.227 

3 

.694 

17338 87206 


4 

.512 

12794 100000 






Loadings 


4»of 



»»of 

Com 

Variance 

Gim4« 

Total 

V ariance 

4* 

39.182 

39.182 

1347 

38.664 

38 664 

30686 

69.868 

1.248 

31304 

®.868 


Scree Plot 



Figure 3. Scree plot for higher order factor analysis 

Internal consistency reliability was computed for these two higher-order construct scales. The first scale. 
Information Seeking was found to have alpha = .71 (respectable according to DeVellis (1991)). Cronbach’s 
Alpha index for the second scale. Information Sharing was .83 (very good per DeVellis’ guidelines (1991)). 
The items forming each higher-order scale are listed in Table 3. 

5.2 Higher Order Factor Analysis and Multidimensional Scaling 

Higher order factor analysis and multidimensional scaling indicated two possible factors which, upon 
examination by researchers, proved to have content (face) validity. One factor consisted of items that were 
related to ICT use for reflection, communication, and sharing. The second factor grouped items related to 
ICT for seeking information related to areas of interest, expertise, or study. Multidimensional scaling was 
conducted to further examine the underlying factors for the ICTL survey. The ALSCAL Euclidian distance 
model with maximum of two dimensions was generated to examine distances and proximities for items in 
relation to one another. Two main output clusters are visible on the Euclidean distance model (Figure 4). This 
two-scale alignment confirms results of higher order factor analysis. Instrument items in quadrant II & III, 
with item #8, which is located near the Y axis were examined as the first of two scales, and all other items in 
quadrants I & IV as the second of two scales which were identified by use of higher order factor analysis and 
multidimensional scaling. This two-factor solution was accepted and resulted in the Information Seeking, and 
Information Sharing scales of the ICTL. 
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Derived Stimulus Configuration 
Euclidean distance model 



Figure 4. Euclidean distance model. Multidimensional scaling plot of ICTL items 


Table 3. Items for higher order scales. Two scales emerged from higher order factor analysis 


ICTL Scale 

# Items 

Item 

Numbers 

Alpha 

Cronbach’s 

Rating (DeVellis) 

ICTL- Total scale 

15 

1-15 

0.77 

Respectable 

subscales: 

Information Seeking 

7 

2,4,7,8,10,13, 

14 

0.71 

Respectable 

Information Sharing 

8 

l,3,5,6,9r,ll, 

12,15 

0.83 

Very Good 


Note: Item #9 was reverse coded. 


5.3 Criterion-related Validity 

Criterion related validity was examined for the three scales of ICTL by correlation analysis with the Mobile 
Learning Scale, a 7-item, unidimensional, Likert-type survey instrument developed by Khaddage and Knezek 
(2011) for used with higher education students. The Mobile Learning Scale version 1.0 was designed to 
assess acceptance of mobile learning, including perceptions of mobile learning devices and tools (Apps) for 
informal learning, and feelings about using theories and models to incorporate mobile learning into higher 
education (Knezek & Khaddage, 2011). Knezek and Khaddage (2011) found the internal consistency 
reliability for the seven-item instrument to be very good (alpha = 0.85) (DeVellis, 1991) among 81 
undergraduate and graduate university students completing the survey in a large Midwestern university in the 
USA during August and September of 2011. A list of the items on the Mobile Learning Scale version 1.0 is 
provided in Figure 5. Correlation analysis revealed concurrent validity between ICTL, Information Seeking, 
Information Sharing and the Mobile Learning Scale. Significant correlations (p < .01) were found between 
Mobile Learning (ML) total scale score and all three ICTL measures: a) ML with ICTL_Total r = .37 (p < 
.0005), b) ML with ICTL_Information Seeking r =.35 (p < .0005), and c) ML with ICTL_Information 
Sharing r =.37 (p < .0005). This magnitude of Pearson Product Moment Correlation would be considered a 
moderate effect size according to guidelines by Cohen (1988). This confirms that the ICTL survey scales 
have demonstrable criterion related validity in the form of alignment with an established Mobile Learning 
Scale. This research finding contributes new knowledge to the emerging set of models and theories 
developing in the field of mobile learning. In particular, the implication is that both information seeking and 
information sharing are important aspects to be considered in mobile learning environments 
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Instructions: Select one level of agreement for each statement to indicate how you feel. 


SD = Strongly Disagree, D = Disagree, U = Undecided, A = Agree, SA = Strongly Agree 



MOBILE LEARNING SCALE 

SD 

D 

u 

A 

SA 

1. 

The rapid development of Mobile Learning devices and tools 
(Apps) has empowered informal learning. 


(2,i 

(3.1 

© 

© 

2. 

Mobile Apps could be integrated seamlessly to support informal 
learning. 


(2) 

© 

© 

(5,i 

3. 

Mobile Apps could bring enormous opportunities into universities 
to further empower informal learning 


(2,i 

© 

© 

© 

4. 

Student acceptance of Mobile Learning in higher education would 
be high. 


l(2) 

© 

© 

© 

5. 

Recent developments in Mobile Learning are leading to the 
exploration of new methods/models at universities 


(2) 

© 

© 

© 

6. 

Theoretical models and methods can assist in informing the design 
for mobile learning Apps 


d> 

© 

© 

© 

7. 

The integration of mobile applications, mobile social networking 
platforms and other mobile technologies has become pervasive in 
teaching and learning. 


(2) 

© 

© 

© 


Figure 5. Mobile Learning Scale (MLS) Version 1.0 items. Note: MLS vl.O 8/201 1 by F. Khaddage & G. Knezek 


Additional associations were found between other attributes of interest and one or more of the scales used 
in this study. Females tended to rate higher than males on mobile learning attitudes (p = .012) but there were 
no significant differences by gender on information seeking or information sharing preferences. Age was 
negatively correlated with information seeking ratings (p = .008) and information sharing (p = .001), but no 
significant correlation was found between age and mobile learning attitude (NS). Mobile learning attitudes 
were positively correlated (p = .002) with self-reported creative tendencies as measured by a 13 item Likert 
Scale. These findings add context to the associations reported in the previous paragraph, and raise additional 
questions to be addressed in future research. 


6. DISCUSSION 

In recent decades the role of formal education as the disseminator of significant learning experiences has 
diminished (Aittola, 1999). Information communications technologies of the twenty-first century support a 
variety of learning, sharing and knowledge acquisition options which can take place in many places 
facilitated by internet-based communications tools (Knezek, Lai, Khaddage & Baker, 2011), thereby altering 
the role of formal education in learning. The information seeking, sharing and mobile access dimensions of 
learning which are measured by validated instruments identified in this paper are well fitted to assist teachers 
and learning technologists to design instruction that will support learning interaction with new options in 
mobile learning technologies and ICT. 

As stated by Naismith, Lonsdale, Vavoula, and Sharpies, (2004, p. 36) the current challenge is for 
educators and technology developers to find a way to ensure that new learning methods are effective - highly 
situated, personal and collaborative for the long term, and accepted by students. Research for formal and 
informal learning in new technology pervasive environments should include data on student preferences that 
can help identify students’ attitudes, level of acceptance, opinions and expectations regarding the integration 
of new technologies such as mobile learning into teaching and learning. This approach is what Andrews and 
Tynan (2012) have referred to as “investigating the human voice” (p. 565) to meet the unique needs of 
today’s learner. Hence, by creating and validating instruments to measure attitudes toward adoption and use 
of new technology options, such as the ones presented in this paper, which can more precisely and efficiently 
assess reliability and validity and make continuous refinements, we can allow for informed understanding of 
ICT learning and information seeking preferences of students. Additionally, the findings of this study imply 
that the ICTL and Mobile Learning Scale are useful for measurement in three areas of focus for the study of 
students’ needs: information seeking preferences and behaviors, information sharing preferences and 
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behaviors, and perceptions of teaching and learning with mobile devices. 

A review of the literature revealed that existing instruments for measurement of technology in teaching 
and learning have frequently focused on self-reported measures that assess student and teacher attitudes and 
proficiency with information and communication technology for technology integration ( Knezek & 
Christensen, 2000). Instruments have also been validated to measure success with teaching and learning in 
Internet enriched learning environments (Hong, Ridzuan, & Kuek, 2003). A number of instruments measure 
different facets of learning. For example, Litzinger, Wise, Lee, & Bjorklund (2003). developed an instrument 
to measure aspects of self-directed learning. The authors contend that there is a need for the ICT and mobile 
learning scales to measure learner preference for ICT and mobile technologies for information behavior such 
as seeking and sharing. 


7. SUMMARY AND CONCLUSIONS 

Data collected online from college student volunteers for a learning preference study was used to refine the 
Information Communication Technology Learning (ICTL) survey. Construct validity was examined in a 
multi-step process. Reliability was analyzed for all 15 items of this survey and found to be respectable (alpha 
= .77) according to established guidelines (DeVellis, 1991). Four possible measurement factors were 
identified by PCA exploratory factor analysis: two of these were considered unacceptable with low internal 
consistency reliabilities according to established guidelines. However, higher order factor analysis and 
multidimensional scaling identified a two higher order constructs with very good and respectable 
measurement properties: Information Sharing (alpha = .83) and Information Seeking (alpha = .71). Scales 
produced from these two factors were found to be significantly correlated (p < .0005) with student mobile 
learning (MLS) attitudes, providing cross-validation evidence for ICTL scales. The correlations between 
ICTL scales and MLS were at the r = .35 level or higher, indicating that sharing and seeking information both 
are perceived as functions to be performed with mobile technologies. These findings support the authors’ 
conclusion that the Information and Communications Technology Learning (ICTL) instrument is worthy of 
further use, either alone or as part of a battery of instruments including others such as the Mobile Learning 
Scale. 

This study also contributes new knowledge to the emerging set of models and theories developing in the 
field of mobile learning. In particular, the implication is that both information seeking and sharing behaviors 
are important points of consideration in mobile learning environments. Further research is needed to 
determine if the instruments used for this research can assist educators and researchers in better 
understanding student preferences for ICT integration in order to improve support for information behavior 
targeting formal and informal learning. Findings from this study, such as the positive correlations identified 
between mobile learning attitudes and creative tendencies, and the confirmed female preference for mobile 
learning while no gender differences regarding information seeking or sharing were found, suggest that the 
ICTL and MLS instruments possess criterion related characteristics that may be useful in addressing 
additional research questions on students’ attitudes towards mobile devices, ICT integration, and information 
seeking behaviors. These areas will be pursued in future studies. 
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ABSTRACT 

This research contributes to answer the question whether learning/cognitive styles of students serve as a justified starting 
point for creating target-group appropriate instruction. The study was realized in a self-regulated problem-based learning 
environment. Data of 56 participants on their individual learning styles, their acquired problem solution and their increase 
in declarative knowledge were collected. Results indicate that a consideration of learning styles to design a matching 
instruction or learning environment does not transfer into higher quality problem solutions or an enhanced increase in 
declarative knowledge. 


KEYWORDS 

Learning style, cognitive style, preflection, prompting, mental models, self-regulated learning 


1. INTRODUCTION 

To be an effective self-regulated learner, students actively have to influence and adjust all learning processes 
of the cognitive, metacognitive, and motivational dimension. Such a multilayered regulation of learning 
enables improvements in procedures of the cognitive, metacognitive, and motivational component of learning 
(Schiefele and Pekrun, 1996, Bransford et al., 2000). Zimmerman’s (2008, 2000, 1998) and Pintrich’s (2000) 
models of self-regulation share those key-components and orchestrate their sub-processes in a 
cyclic-reflective understanding to specific phases of self-regulated learning. 

Learning styles are regarded as one of the crucial factors to be taken into account when designing 
instruction and learning environments: “Instruction designed to address a broad spectrum of learning styles 
has consistently proved to be more effective than traditional instruction, which focuses on a narrow range of 
styles” (Felder and Brent, 2005, p. 59). However, the scientific understanding of the concept of learning style 
is rather inconsistent. Learning-style models range from holistic approaches (e.g., the Dunn and Dunn 
learning-style model; Dunn et al., 2009) to specific models that focus on certain dimensions of learning, i.e., 
information gathering, processing, and retrieval (e.g., the Felder-Silverman learning-style model; Felder and 
Silverman, 1988). 


2. THEORETICAL BACKGROUND 

2.1 Learning Styles in Theory and Educational Practice 

The field of educational psychology comprises numerous learning style models, all aiming at an 
improvement of learning. Learning style assessments could contribute to the learner’s intrapersonal 
knowledge as a part of his/her declarative meta-knowledge. Accordingly, intrapersonal knowledge is defined 
as knowledge about the own thinking, memory, and corresponding tendencies (Brown, 1984). Therefore, by 
knowing about specific preferences a learner unifies, he/she at least fulfills a basic prerequisite to self- 
regulate his/her learning process and/or environment in a way that suits his/her individual learning 
preferences. 
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However, assessing learning styles could not only contribute to the learners’ meta-knowledge and 
subsequently create the basis for improvement of his/her control mechanisms in a self-regulated learning 
environment. Even before learners participate in a (self-regulated) learning environment, learning styles 
could be an important factor. As aforementioned, several ID-models advise to assess the actual learning 
styles of the target-group as a central aspect of the learners’ characteristics (e.g., Morrison et al., 2011, 
Rothwell and Kazanas, 2008). Analogous to self-regulation mechanisms of learners that can occur in the 
phases of self-regulated learning, instructional designers could consider the predominant learning styles of 
their target-group when designing instruction. Theoretically this has to result in an optimization of the 
designed instruction by improving its suitability for the target group/learner. 

The relevance of learning styles for instructional designers as well as the field of self-regulated learning 
and according research, thus, seems to be unquestionable. However, a vast number of different 
understandings of learning style definitions, theoretical positions, models, and measures can be identified. 
This leads to an increasing lack of clarity what a learning style is and how it facilitates instruction and ( self- 
regulated) learning processes. Cassidy (2004) states: “As a consequence of the quantity of research, the 
disciplines and domains in which the research is conducted, and the varied aims of the research, the topic has 
become fragmented and disparate” (p. 419). To deal with learning styles, first, a basic understanding for the 
concept is obligatory and to be presented in order of not to contribute to further dilution of the concept. 

Hawk and Shah (2007) consider learning styles not as a state. The authors mention that learning styles 
and learning style models all share the basic assumptions that “students learn in different ways”, the concept 
of “learning style is a component of the wider concept of personality”, and “learning style falls into the 
categories of dispositional traits and characteristic adaptations where there are differences across individual 
humans but there are groupings of humans who have common or similar learning style characteristics” 
(Hawk and Shah, 2007, p. 2). 

Referring to Hawk and Shah's (2007) characterizations as a basic premise, an interchangeable use of the 
terms “learning style” and "cognitive style” seems to be possible. Some models comprise solely cognitive 
aspects of gathering, processing and retrieving information (e.g., the Felder-S Overman model), whereas other 
models include constructs dependent on the age-appropriate assessments administered, e.g., environmental 
stimuli (sound, light, temperature, seating, etc.) that feature no direct relation to cognitive processes (Dunn et 
al., 2009). Riding and Cheema (1991) further elaborate on the not well defined usage of learning style and 
cognitive style: “Whilst cognitive style is a bipolar dimension, learning style entails many elements and are 
usually not ‘either-or’ extremes. One either has or does not have the element in one's style, similarly, the 
absence of one element does not necessarily imply the presence of the opposite element” (p. 194). 

2.1.1 Empirical Findings on Learning Styles 

Although, terminological shortcomings in leaming/cognitive style research have been made explicit for 
several decades and the benefit of taking such styles into account when designing instruction seems to be 
obvious - at least from a theoretical perspective - scientists still argue about its practical efficacy. Regarding 
various empirical studies on learning styles, results still draw an unclear picture about the “true” relevance of 
the concept. 

An in-depth literature review revealed numerous contradicting results of research on learning styles. The 
following investigations and reviews all support the benefit that arises from considering learning styles when 
designing instruction (e.g., Dunn et al., 2009, Schmeck, 1988, Hayes and Allison, 1993, Hayes and Allison, 
1996, Riding and Grimley, 1999, Bajraktarevic et al., 2003). The scope of the instructions observed in these 
studies provides a wide range from traditional training situations to hypermedia learning environments. In 
contrast, other studies and reviews disagree with the aforementioned (e.g., Kavale and Forness, 1987, Snider, 
1992, Cook et al., 2007, Gilbert and Swainer, 2008, Martin, 2010). Analogous to the contradicting studies, 
the investigated learning environments and instructional settings also range from traditional classroom to 
multimedia learning environments. Thus, as already mentioned, the picture about the “true” relevance of 
learning styles remains unclear. Positive effects of taking learning styles into account when designing 
instruction are still questionably. 

2.1.2 Index of Learning Styles (ILS) 

The Index of Learning Styles (ILS; Felder and Silverman, 1988) was “designed to capture the most important 
learning style differences among engineering students and provide a good basis for engineering instructors to 
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formulate a teaching approach that addresses the learning needs of all students” (Felder and Spurlin, 2005, p. 
103). The model shares commonalities with other popular learning style approaches, e.g., the Learning Style 
Inventory (LSI; Kolb, 1984) or the Myers-Briggs Type Indicator (MBTI; Lawrence, 1994). 

With regard to the theoretical assumptions made above, the ILS is explicitly said not to include ‘either-or 
categories’ of its bipolar dimensions. All scales are to be understood as continua, which means that a 
student’s cognitive preference to learning on a given ILS scale may be either fairly well balanced, 
moderately, or strongly distinctive for one or the other pole of the scale. The four bipolar ILS dimensions can 
be described as follows (Felder and Silverman, 1988): 

• Active - Reflective. Active learners tend to gather and understand information best if they engage 
with it actively and try things out, e.g., by debating, bringing something to application, or via 
teaching-back. Reflective learners prefer to think about new things respectively information for 
themselves first. The motto of active learners is “Let us try and see how it works”, whilst reflective 
learners pursue the principle “Let me first think carefully about it”. 

• Sensual - Intuitive. Sensing learners tend to do well in learning facts and to follow established 
approaches and procedures when solving problems. They are more goal-oriented, carefully and 
patiently, but avoid complex issues or surprises. Intuitive learners on the other hand prefer to 
explore different possibilities and relationships and innovative approaches. The can better grasp new 
concepts, work usually faster and more innovative and have less difficulty with abstract concepts 
and mathematical expressions. However, they tend to avoid rote learning, repetition, routines and 
fixed schemes. 

• Visual - Verbal. Visual learners remember more of what they have seen, e.g. pictures, diagrams, 
flow charts, films, demonstrations, etc. Instead, verbal learners prefer linguistically based learning 
that is written and spoken information or declarations. However, it should be noted at this point, that 
regardless of the deviation of a person, the combination of visual and verbal information is most 
conductive. 

• Sequential - Global. Sequential learners tend to understand better by learning in logical linear 
steps, where each step is the logical consequence of the previous step. In contrast, global learners 
rather tend to make big steps and gather different material and information quasi-random and 
without the recognition of contexts and relationships, but suddenly, they understand the whole 
context. 

2.2 Designing Model-Based Learning Environments 

2.2.1 Mental Models as a Basis for Analyzing Problem Solutions 

A mental model is an idiosyncratic representation of a fact or a thing, of ideas or more generally an ideational 
framework about something interesting in the world. Mental models, as types of representations, rely on 
language and use symbolic pieces and processes of knowledge to construct a heuristic for a situation. The 
theory of mental models follows the constructivist assumption that human knowledge is always fragmentary 
and thus regarded as dynamic, because it is constantly expanded and/or modified (Johnson-Laird, 1983, 
Pirnay-Dummer et al., 2012). Hence, mental models function as internal models of the outside world and 
produce subjective plausibility (Ifenthaler and Seel, 2011). 

With regard to problem solving processes, studies show a strong correlation between effective mental 
modeling and the solution of complex problems (Jacobson, 2000). According to Dorner (1976) problems are 
characterized by means of three components; 1.) an unsatisfying actual state, 2.) a desired target state, and 3.) 
a barrier which currently inhibits the mental transformation of the actual to the target state. The bigger this 
barrier is the more complex the problem is regarded. The definition of problem solving processes clarifies the 
importance of mental models for successful problem solving. Funke (2003) states that problem-solving 
thinking is characterized by filling the gaps within an action plan that cannot be accomplished routinely. For 
that purpose a mental model is constructed which bridges the way from the initial to the goal state (Ifenthaler 
and Seel, 2011). 

Ifenthaler (2010, 2008) describes the externalization of internal cognitive structures as a deliberate 
communication process of mental models. Whenever the observation of knowledge representation is 
essential, externalization is necessary, because direct access is not possible. Accordingly, methods to measure 
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knowledge representations result from the conscious communication of mental models, e.g., via thinking 
aloud, text writing, constructing graphs, knowledge or concept maps (Ifenthaler, 2010). This results in the 
differentiation between internal knowledge representations and external, so called re-representations 
(Ifenthaler, 2010). The re-representations are communicated on the basis of the mental representations and 
adequate sign and symbol systems (Shute et al., 2009, Seel, 1999). Accordingly, the solution to a 
phenomenon in question which is represented by an individual mental model, and consecutively the re- 
representation of this model as a result of a communication process allows the investigation of individual 
problem solutions (Ifenthaler, 2010). 

2.2.2 Preflection - a Pendant to Reflection 

Expert self-regulated learners are said to be effective in actively influencing and adjusting all learning 
processes of the three key-components of learning: cognition, metacognition, and motivation (Schiefele and 
Pekrun, 1996, Bransford et al., 2000). Metacognitive processes are attributed to play a superior role, because 
metacognition can serve as a tool to regulate the further dimensions of learning and consequently optimize 
their specific aspects with regard to the goals set (Leutner and Leopold, 2003). 

Zimmerman's model of self-regulation (2008, 2000, 1998) suggests a cyclic-reflective understanding of 
self-regulation. Three phases are identified analogous to an input-output-system, where each result of an 
action sequence affects the following. At the beginning of a learning cycle stands the forethought phase. 
Pintrich (2000) adds “planning and activation” to the label of that initial stage of each self-regulated learning 
cycle. In general, it can be characterized as preactional phase, followed by the actional and post-actional 
phase. Consequently, the learning phase of action is enclosed by a preparatory and a post-processing phase. 
In terms of metacognition, the differentiation between prospective and retrospective monitoring (Nelson and 
Narens, 1994) can be applied to the surrounding phases, represented by two psychological constructs: 
Reflection is primarily characterized to refer to something experienced (Pintrich, 2000), thus suiting the 
postactional phase, whereas preflection is relating to future events and based on prospective thoughts, thus 
suiting the preactional phase. The latter, hence, is to be understood as the pendant to reflection. In regard to 
what Dewey (1933) described when introducing the concept of reflection - humans learn more from 
reflection on their own experience than from the experience itself - preflection can be characterized as 
reflection's inverted counterpart: a tool for optimizing subsequent experience, respectively problem-solving 
and learning processes. Accordingly, preflection involves the activation of central learning relevant structures 
(e.g., content knowledge, strategies, values, interest, etc.) that are prospectively regarded to be beneficial for 
future performance. 

2.2.3 Prompts as an Instructional Intervention in Self-Regulated Learning Environments 

External instructional interventions have to be used at the right time and the right extent, so that the self- 
regulatory learning process is not disturbed, but rather supported. Instructional designers need to develop 
learning environments implementing an appropriate level of external- and self-control which poses not an 
easy challenge. Therefore, research has gained big interest in investigations of demand-oriented support and 
assistance interventions ( scaffolds ). One specific form of scaffolds are formed by instructional cues, so called 
prompts. Within the context of self-regulated learning they are regarded as effective, because they serve as a 
“short-time intervention” (Bannert, 2009) and only represent a minimal external control mechanism. 

Davis (2003) and Ifenthaler (2012) differentiate between two forms of prompts: generic and directed. 
Generic prompts follow the principle “stop and think”. A generic prompt encourages learners to interrupt the 
current learning or problem solving processes for a moment and reflect. Here, the object of reflection is left 
completely open. There are no specific issues highlighted or instructed by the prompt. 

Directed prompts, on the other hand, follow the principle “stop and think about ...”. Davis (2003) states 
that the more specific directed prompts should be more effective than general, unspecific prompts, if they 
were adequately implemented in the learning or problem-solving environment. It is assumed here that 
individuals who do not have the necessary knowledge and skills required for the desired deliberations - 
reflective or preflective - need an additional instruction along with the prompt. Directed prompts include 
such an instruction, for example, in form of to complete sentences. Ifenthaler (2012) alternatively found 
generic prompts to be more efficient compared to directed prompts, because they leave a certain amount of 
autonomy for self-regulative acting. Still, Ifenthaler (2012) agrees on Davis’ (2003) argument that directed 
prompts may be more efficient for learners that do not already have a specific set prior knowledge and skills. 
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2.3 Research Questions and Hypotheses 

The contrasting views within the community of educational psychology as well as the contradictory empirical 
results on the effects of taking learning styles into consideration when designing instruction led us to the 
present study. The objective is to investigate the influence of students’ learning styles on the effectiveness of 
instructional interventions (metacognitive prompts within a problem-based learning environment). Regarding 
the theoretical foundation of learning styles, we assume that reflective students as well as students with a 
propensity for verbal learning develop higher quality problem solutions and perform better in retrieving 
declarative knowledge than the active type of students and students with a preference for visual learning. The 
reasons for these assumption are that (1) active learners tend to profit from actively doing something, e.g., 
discussing, applying information, teaching back, or trying it out; reflective learners gain much more from 
introspective processing and thoroughly thinking something through before trying it out (Kolb, 1984) and (2) 
the fact that the learning environments’ metacognitive prompts plus the problem solving itself happen 
basically verbal. Thus, the following hypotheses are addressed: The more reflective and verbal the learners 
are, the higher the quality of their developed problem solutions (Hypothesis 1). The more reflective and 
verbal the learners are, the better they perform in retrieving declarative knowledge (Hypothesis 2). 


3. METHOD 

3.1 Participants and Design 

A total of N = 56 undergraduate students (42 female and 14 male) from a German university took part in the 
experiment. Their average age was 22.73 years (SD = 3.83). They were all enrolled in an introductory course 
on research methods and were asked to participate in an experiment to meet the course requirements. 
Participants were randomly assigned to two experimental groups (GP = generic prompt, = 28; DP = 
directed prompt, n 2 = 28). Participants in the GP group received a general prompt to conduct a phase of 
preflection on the subsequent problem-solving task. Participants in the DP group received instructional to- 
complete-sentences in order to prompt preflective thoughts. 

3.2 Materials 

3.2.1 Problem Scenario and Prompts 

The prompts were embedded in a problem-based self-regulated learning environment where participants were 
asked to help their mother who was moaning about dorsalgia and concerned about having a herniated disk. A 
partially illustrated article on the spine’s and spinal cord's anatomy and functionalities as well as the reflex 
circuit was used as learning content. 

Generic and a directed prompts were developed accordingly. The generic prompt included a general 
suggestion for a preactional phase of forethoughts, planning, and activation on the subsequent problem 
solving process and according self-regulation mechanisms ( please use the following ten minutes to optimize 
your upcoming problem solving process through well-thought-out planning and preparation). In contrast, the 
directed prompt was designed more specific. In total it contained eleven to-complete-sentences in order to 
induce the specific processes of prospective considerations and preactional self-regulation (e.g., activation of 
prior knowledge: About the substantive subject area of the problem I already know that...; goal setting: To 
achieve the main objective I’ll set the following sub goals...). 

3.2.2 Domain Specific Knowledge Test 

The knowledge test included eleven multiple-choice questions with four possible solutions each (1 correct, 3 
incorrect). Two versions (pre- and post-test) of the domain-specific knowledge test were administered (in 
which the eleven multiple-choice questions appeared in a different order). It took about eight minutes to 
complete the test. 
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3.2.3 ILS 

A German translation of the Index of Learning Styles (ILS; Felder and Spurlin, 2005) was used to assess the 
students’ individual learning styles. Each of the four ILS dimensions consists of eleven items. Each item 
possesses two response possibilities ( a or b) that either correspond to one extreme of a dimension's 
continuum (e.g., active vs. reflective). Felder and Spurlin (2005) report correlation coefficients for the ILS- 
scales of three test-retest studies with values between .72 and .87 at a four weeks interval, .60 and .78 at 
seven months, and .51 and .68 at eight months. 

Regarding the validity of the ILS, it is to report that both exploratory factor analysis with an eight-factor 
solution, as well as a feedback analysis of the sample with approval levels of 80 percent for the Sequential- 
Global scale and over 90 percent for the economies of scale Active-Reflective, Sensing-Intuitive, Visual- 
Verbal seem to prove the validity of the ILS (Litzinger et al., 2007). 

3.3 Procedure 

First the participants completed the domain-specific knowledge test, a demographic data and the ILS 
questionnaire. Then they were introduced to the problem-scenario and the groups received their specific 
prompts to preflect. The learning material was handed out 10 minutes afterwards. Participants worked for 30 
minutes on the solution of the problem. After a total of 40 minutes for solving the problem, including the 
induced phase of preflection, participants were asked to represent their solution of the problem as a written 
text. Finally, the post-version of the domain-specific knowledge test was completed. 

3.4 Analysis 

3.4.1 HIMATT 

HIMATT (Highly integrated Model Assessment Technology and Tools; Pirnay-Dummer et al., 2010) allows 
an automated analysis of the text-based problem solutions. A detailed description of the seven measures of 
HIMATT, which include four structural and three semantic measures, is provided by Ifenthaler (2010). High 
reliability and validity measures have been reported for HIMATT ( Pimay-Dummer et al., 2010). 

3.4.2 Reference Model 

In order to quantify the qualities of the participants’ problem solutions an expert solution was used as a 
reference model for our statistical analysis. The text-based reference model was worked out within the same 
problem-based learning environment by an orthopedic specialist and apprenticed physiotherapist. 


4. RESULTS 

Table 1 describes the distribution of learning style tendencies of all four ILS-scales for the 56 participants of 
our experiment. The distribution is in line with reports from other studies that used the German translation 
(Derntl and Graf, 2009) as well as the English version of the ILS (Felder and Spurlin, 2005). 


Table 1. Distribution of learning style preferences 


active 

reflective 

sensing 

intuitive 

visual 

verbal 

sequential 

global 

36 

20 

39 

17 

47 

9 

28 

28 

64.3 % 

35.7 % 

69.6 % 

30.4 % 

83.9 % 

16.1 % 

50% 

50% 


On the declarative knowledge test, participants could score a maximum of eleven correct answers. They 
scored an average of M = 3.23 correct answers (SD = 1.53) in the pretest and an average of M - 6.16 correct 
answers (SD = 1.68) in the posttest. The increase in correct answers (not normally distributed) was 
significant, Z = -6.049, p < .001, r = -.808 (strong effect). 

For testing our hypotheses, we computed an ILS main score for each participant using the according 
numeric relatives of both hypotheses-relevant ILS-scales, active -reflective and visual-verbal. Then, 
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experimental groups were segmented into three sub-groups via percentiles (GP: active-visual n Ia = 9, non- 
distinct n Ib = 4, reflective -verbal n lc = 16; DP: active-visual n 2a = 17, non-distinct n 2b = 0, reflective -verbal 
n 2c = 11). To check hypothesis 1, the qualities of problem solutions were investigated by independent t-tests 
respectively Mann-Whitney-tests, if HIMATT measures were not normally distributed. 

For the GP group t-tests revealed no significant effect for GRM, f(23) = .424, p = .675; STM, f(23) = 
1.344, p = .192; GAM, t( 23) = 1.463, p = .157; PPM, t( 23) = -.729, p = .473. Additionally, the Mann- 
Whitney-tests showed no significant effect for SFM, U = 60.5, p = .514; CCM, U = 63.0, p = .610; BPM, U = 
63.0, p = .610. For the DP group t-tests revealed no significant effect for SFM, f(26) = -.391, p = .699; GAM, 
K 26) = -1.046, p = .305; CCM, t( 26) = -.445, p = .660; PPM, t( 26) = -,100,p = .921; BPM, t( 26) = -.365, p = 
.718. Additionally, the Mann-Whitney-tests showed no significant effect for GRM, U = 68.0, p = .225; STM, 
U = 64.5, p = .172. Accordingly, no significant differences of the quality of problem solutions between 
reflective-verbal and active-visual learners could be identified in either experimental group. Therefore, 
hypothesis 1 is rejected. 

Secondly, we assumed that the increase in declarative knowledge for subjects with a preference for 
reflective and verbal learning is superior compared to others. For the GP group (knowledge increase not 
normally distributed) the Mann- Whitney-test revealed no significant differences between reflective-verbal 
and active-visual learners, U = 51.5, p = .238. For the DP group (knowledge increase normally distributed) t- 
test revealed no significant differences between reflective-verbal and active-visual learners, t{ 26) = - 1 .083, p 
= .289. Accordingly, we reject hypothesis 2. 


5. DISCUSSION 

A consideration of the orientation of the applied HIMATT measures (structural vs. semantic) shows that in 
average the subject performed ways better on the structural than on the semantic level of the posed problem. 
This suggests that, although a minimum of necessary complexity could be obtained, the subjects were not 
able to use sufficiently correct concepts as well as to create semantically correct relationships within their 
solution to the problem. For higher quality results they should have increased their usage of correct concepts 
and propositions. 

Contrary to our hypotheses, the more reflective and verbal study groups did not differ significantly from 
the more active and visual subjects, neither in the quality of their developed problem solutions nor in their 
increase in declarative knowledge. Accordingly, we conclude that the matching of our instructional scaffolds 
respectively both the suiting generic and directed prompts did not work beneficially for their target group. 
Therefore, either the prompts or the dimensions of the Felder-Silverman learning style model have to be 
questioned in terms of validity. 

Regarding the first two critical aspects, we have to mention that our experimental design with its strict 
time schedule could be one reason for the fact that the consideration of learning styles did not transfer into 
higher quality problem solutions and an increase in declarative knowledge. But at the same time, one could 
assume that a matching respectively mismatching between the type of the developed self-regulated problem- 
solving environment as well as the embedded instructional support (generic/directed prompts) on the one 
hand, and the present learning styles (either more distinctive reflective-verbal or active-visual) on the other 
hand, still would result in differences regarding our dependent variables. 


6. CONCLUSION 

Given that generic and directed prompts have been found to be an important instructional method for 
self-regulated learning in problem-solving environments (Ifenthaler, 2012, Davis, 2003, Wirth, 2009), we put 
in question if learners really benefit from taking learning/cognitive styles into account when designing 
instruction. Based on our empirical finding, one would have to answer »No« to that question. 
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ABSTRACT 

Assessing students in online university courses poses challenges to the reliability factor of the measures being utilized. 
Some programs have the latitude to incorporate proctored assessments, but this is not always practical in asynchronously 
structured courses reaching out across a broad geographic region. This paper explores digital audio and video outputs as 
viable student pontification alternatives. 
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1. INTRODUCTION 

Assessment is the process of collecting, synthesizing, and interpreting information in order to make a 
decision (Airasian & Russell, 2007). Value in any instructional system comes from assessment: what is 
assessed in a course or a program is generally associated with value; what is valued becomes the focus of 
activity (Swan et al., 2007). Effective assessment typically includes ongoing “formative assessment'’ 
checkpoints, and end of term “summative assessment.’’ Instructors signal what knowledge skills and 
behaviors they believe are most important by assessing them, while students quickly respond by focusing 
their learning accordingly (Swan et al., 2007). The end-of-course assessment method, and more specifically 
the requirements that underlie this assessment mode, make a difference (Struyven et al., 2006). In an online, 
asynchronous course, whereby the students and instructor do not meet, obtaining reliable assessment 
measures becomes more difficult than in a traditional face-to-face class. If exam proctoring is not an option 
other innovative means of measuring student learning outcome must be examined. This paper explores the 
creation of digital audio and video as a means for students to demonstrate learning outcomes in online 
university courses when exam proctoring is not possible. Although it is possible as an instructor to elicit 
online quizzes, papers, and projects from students it is important to collect several pieces of information 
about the performance being assessed to increase reliability (Airasian & Russell, 2007). Without the exam 
proctor possibility, determining who is (and how many are) involved in the submission of the common 
assessment items is problematic. As a means to strengthen assessment reliability, and foster students’ creative 
engagement, the use of digital audio and video pontifications in online courses at a large USA university will 
be examined and discussed as a viable means to foster more reliable assessment outcomes for students and 
instructors. 


2. ASSESSMENT 

In any course measuring the growth students have made towards the objectives is critical to determine the 
effectiveness of instruction. As instructors we have to be certain that our efforts are resulting in optimum 
outcomes for students. In higher education written exams are often the assessment means of choice due to 
large numbers of students and limited instructor time. Final course grades and exams are the most common 
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measures of learning outcomes for seniors across Majors (NSSE, 2010). Scheduled test events tend to 
increase students’ study time efficiency (McKenzie, 1979). 

2.1 Types 

There are many categorizations of assessment. First it is possible to make a distinction between assessment 
and test. The first being the process of determining the learning gains, and the second being the instrument or 
measurement tool for gathering the data. Some use the terms evaluation and assessment in a similar manner. 
It is important to consider the time-based snapshots of learning which are addressed by on-going incremental 
measurements (formative), and culminating end of term measurement of the entire term (summative). When 
looking at the instrument for gathering measures of learning itself there are two achievement test output 
categories to consider: written (conceptual, selected response), and performance (applied, task-oriented). 
Constructed response tests (short answer, essay), on the other hand, blur the boundaries between written and 
performance. They are written, and sometimes performance. “Like stories, reports, or show-your-work 
problems, essays and extended-response test items are important forms of performance assessments (Airasian 
& Russell, 2007). Educators’ perspective on what constitutes performance varies, but the existence of three 
common characteristics prevails in identifying assessment as performance: (a) multiple evaluative criteria; 
(b) pre-specified quality standards; and (c) judgmental appraisal (Popham, 2011). Bloom’s taxonomy of 
instructional objectives correlates closely to performance when students are expected to recall and apply 
information in an actual task. In some cases schools are requiring student exhibitions, culminating projects, 
experiments, solving of realistic math problems, and various other demonstrations of competence (Slavin, 
2012 ). 

2.2 Online Options 

In online courses the assessment options are impacted by the student population demographics. Although 
students take online courses frequently at the campus of which they are a resident due to schedule restraints, 
remote students pose the greatest assessment challenge. Meeting with students online in videoconferencing is 
one way instructors assess students informally through interactive dialogue. Oftentimes students indicate 
dissatisfaction when instructors of online courses offer them synchronously at scheduled times due to their 
time/place -bound circumstances. If a student is beyond a reasonable commuting distance or has set hours of 
employment it is difficult to attend any scheduled class whether face-to-face or online. In the asynchronous 
delivery scenario the time can be more forgiving, but the options for assessment are more limited. It is 
possible to develop selected response (multiple-choice, true/false) and constructed response tests in course 
management systems, although reliability is an issue. 

2.3 Reliability 

When circumstances allow having a test proctored for online students will increase its reliability. Reliability 
in this instance refers to consistency over time. If an instructor were able to administer a test to the same 
student at different times ideally the results would be the same. When a test is administered, aspects related to 
the test construction itself, the student, graders, and various circumstances surrounding its administration 
could cause the results to be inconsistent (Slavin, 2012). One major factor that can affect the reliability of a 
non-proctored online exam is its equivalence to a take-home or open book test. In a face to face or proctored 
scenario the test-taker is being monitored. With the take-home scenario used in standard face-to-face classes 
it is possible to have supplemental in-person exams for context. Furthermore, the instructor has constant 
in-person engagement with the students which can provide an opportunity for oral engagement on the subject 
matter. In the fully online situation, however, it is difficult to gauge who or how many are working on the 
same exam. Given such test reliability is compromised. In one study (Watson & Sottile, 2010) students 
reported equitable rates of cheating in face-to-face classes as compared to online classes, but 5.2% more had 
someone else give them answers during an online class quiz or test. It is possible students perceive this as 
acceptable collaboration, not cheating. 
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3. THE COURSE 

The events of this study transpired in a semester-based Technology for Educators course. It is organized in 
accordance with the Quality Matters guidelines for online instruction, and by particular themes: (a) Getting 
started; (b) The Internet; (c) Computer-based software; (d) Audio; (e) Interactive hypermedia; and (f) 
Multimedia. 

3.1 Participants 

In the spirit of action research “to address an actual problem in an educational setting. . ..practical issues that 
will have immediate benefits for education” (Creswell, 2008) an online Technology for Educators class was 
analyzed in progress to understand assessment options in the online class scenario better. The course is 
required for all pre-service teachers in the Elementary Education program of a large southern USA 
university. One online section (N=21) was analyzed in-depth by the researcher within the usual scope of its 
delivery. The students are comprised of non -traditional older students, primarily female, and 24% Hispanic. 
The Elementary Education Program is a transfer program; therefore students join the program in their third 
year of a bachelor's degree, and with a foundational development in elementary education. 

3.2 Procedures 

As the basis of the content in this paper, the researcher analyzed methods for students to demonstrate 
competency of learning objective while increasing the reliability of assessment. The analysis is based upon 
the formative and summative project outcomes of students in the undergraduate Technology for Educators 
courses in a teacher education program. For the final exam students are given a choice of either writing a 
paper, or developing a comprehensive video. If they choose the video they may create it as either an 
individual or group project (self-selected groups). There are also other key assignments throughout the 
semester that have students expound upon their growth in the course content through various other digital 
outputs that incorporate text, static images, audio, video, or a combination. Group projects have the means to 
provide increased student understanding of content and instructor related advantages including multiple 
perspectives and pooled efforts (Young & Henquinet, 2000). From an instructor’s standpoint, presentations 
provide an alternative means for students to demonstrate their competency vested in a culminating course 
project (Arnold. 2010). As a means to capture the “presentation” component of a face-to-face class in the 
online course medium the final project had the presentation element at its core. The process of presenting acts 
as reinforcement for learning that will oftentimes motivate presenters towards adequate preparation and 
information grounding (Arnold. 2010). Students are able to demonstrate meaningful, multidimensional tasks 
via this authentic assessment (Montgomery, 2002). The course at the heart of this study. Technology for 
Educators, was broken down into five modules, each with three weeks devoted to a certain theme. Given that 
the course is primarily for pre-service teachers the focus was on pedagogy and using technology to support 
the standards-based subjects in the classroom. Theoretically and practically, teaching requires substantive 
merging of content, pedagogy, and technology knowledge (Roblyer & Doering, 2010). Each of the modules 
had an overlying technology-based theme with multiple technologies addressed, substantial readings, 
academic content standards tie-in, pedagogical foundation, and emphasis on integration. 

3.3 Technology Integration 

Over time computer programs and online applications wax and wane. Many withstand the test of time, but 
undergo significant repackaging. Given the continual transitioning of computer technology the focus of the 
Technology for Educator courses discussed in this paper has been on helping students become more 
confident in exploring technology while focusing on “why” we need and choose technologies over “what” 
and “how.” For instance the first module’s theme is “online.” In this theme students explore the topics digital 
information integrity, information literacy and multiple issues pertaining to online information exchange: 
legalities, safety, equity, societal, and reliability. During each module students use and create comprehensive 
projects with multiple cloud and computer based technologies while exploring an instructional 
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delivery/e-leaming concept such as podcasting. Their culminating project for this module, as an example, 
would result in a precise public service announcement embedded in a Wiki page and uploaded to an RSS 
feed site (iTunes U), which demonstrates students’ growth (or lack of) from the multiple readings (textbook 
& journal articles), instructor created multimedia presentations, and digital technology explorations. At each 
step of the way students are reminded to critically analyze through a K-12 teacher’s lens. 

3.4 Input and Output 

It is possible to look at the process of developing one's content knowledge as “Input” and demonstrating 
what one knows as “output.” More commonly this is referred to as learning and assessment. Hunter (2004) 
equated the terms with input of information into the students’ cognitive learning processes, and output of 
information in a mastery of the learning objective sense so that proper assessment may occur. Output is also 
associated with the active process of learning whereby the process of output draws heavily upon the content 
knowledge students experienced through the input process reinforcing the learning (Arnold & Moshchenko, 
2009). When students are given the opportunity to produce a tangible product or demonstrate something to an 
audience, their willingness to put forth quality increases (McTighe, 1996). Given the complexities of reliable 
assessment in the online environment, and that the students of the Technology for Educators course were 
taught the skills to create podcasts and video-based digital stories alternate assessments termed Technology 
Integration Pontifications were developed to analyze student growth in the course subject-matter. 

3.5 Performance Pontification 

Digital video editing is well suited for providing authentic, meaningful reflective experiences for teachers 
(Calandra et al., 2009). If it is more pointed in its output with specific criteria then it becomes a viable 
assessment tool. When constructed by the students who are being assessed, and as participants in the video, 
the instructor will be able to analyze the video for key levels of pontification pertaining to the course and 
assessment objectives as the following pontification assignment summary illustrates. 

In this final project you will create a fun digital story on a subject of interest that would be covered 
in an elementary classroom and that supports a content area. It may be anything from science (rocket 
propulsion for instance) to a social message (wash your hands frequently to reduce germs), or any 
other subject you would expect students of your favorite grade-level to learn (look in the content 
standards for a grade-level and subject of choice to identify a specific Performance Objective). In 
the spirit of the “reality” TV mash-ups (i.e., “Survivor” for instance where the program shows the 
tribes in action then cuts away to an individual sharing his/her perspective on that action in an 
interview scenario) I would like you to intersperse yourself into the video as the teacher giving your 
perspective regarding the use of technology in the leaming/teaching process & with your chosen 
subject, while teaching the viewer about a chosen topic (i.e., the earth's rotation/tilt & seasons). 
Have fun with this and get kids involved if possible! You may do this as a group project with other 
students. (S. Arnold, personal communication, April 23, 2012) 

In order to discern specific concepts critical to the outcomes of the course further detailed criteria were 
included. Some pertained to the technology skills whereas others were targeting educational technology and 
integration with the elementary school subject-matter concepts. The following are abbreviated samples of 
measurable objectives included in the project: 

• Refer to and include specific Educational Technology supportive content from at least 6 journal 
article or textbook sources that were assigned during the term 

• Devote about 1/3 of your video to talking about integrating technology into the classroom, and the 
remainder to teaching about a specific topic in a grade-level and subject of choice. Be sure to 
combine them so it does not appear like two separate videos. 

• Include at least 3 separate motion video clips of yourself talking about integrating technology into 
teaching & learning. 

• Include at least 2 separate audio clips of yourself talking about images, 3 rd party motion video clips, 
technology integration, or explaining visual examples of the subject matter. 

• Connect with and identify multiple standards: Information literacy, NETS*T, state academic content 
standards, and state educational technology standards for students. 
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• Make the presence of each group member equitable and evident throughout the video 

• Demonstrates competency with the following technologies/technology processes: Movie Maker, 
Audacity, online file conversion, iTunes, YouTube, ID Tag Editor, synchronized and overlapping 
soundtrack and narrations. 

• Effective integration of still images, motion video, text slides, overlays, soundtracks, and narrations. 

• Important elements of a presentation: Introduction, body, and conclusion 

On a smaller scale, and in a similar manner students were given a reading response assignment whereby 
they had to create an audio narrated hypermedia presentation (PowerPoint) in which they identified key 
points made in the readings as text on the slides and discussed them in audio format. Having met with each 
student individually at the beginning of the semester in a videoconferencing site, and requiring students to 
post audio introductions in their e-portfolios the instructor was familiar with students’ voices. Given such it 
resulted in more personable assessment than written papers. A couple of drawbacks, especially pertaining to 
use in the online environment, include devising a systematically reliable video evaluation means, and the 
technological requirements for developing a video that represents one's content development. Validity is also 
important to this type of assessment; does the video allow an instructor to measure the conceptual knowledge 
that needs to be measured? Identifying the expected outcomes wasn’t really a problem, but some students 
opted to read from scripts which can leave the evaluator wondering if the presenter is engaged or simply 
reading information that is not internalized. 

3.6 Evaluating Audio & Video 

From the grading perspective there is still a disparity in reliability from one instructor to another. Multiple 
teachers grading the same essay paper will assign grades ranging from A to F with some teachers making few 
to no comments or marks on the papers, but instead just producing a grade (Brimi, 2011). As noted above 
there is a convergence of technology use skills, technology integration with subject-matter propensity, and 
any given number of subtopics pertaining to educational technology covered in the course that must be 
weighed when evaluating a video produced by students of a Technology for Educators course. Time is a 
critical element in the analysis, but quality is the most decisive in determining if students are pontificating 
about the concepts covered in the course. In the scope of this analysis most students were able to expound 
upon their chosen topic (a science lesson on volcanoes for instance), but most commonly underdeveloped 
their connection of the lesson topic to their use of technology to demonstrate it, or other examples of 
technology that would further support the teaching of the lesson. The next confounding factor that tended to 
affect students results, whether audio or multimedia, was the technology medium being used for the output. 

3.7 Technological Factors 

Early in the course the technology skillsets were more limiting to the quality of course concept-infused 
outputs than later in the term. Given such the course was structured with less complex technological 
components in the beginning; create a Wiki site via a Web content management system for instance. 
Week-by-week new technologies are introduced. During the first three -week module students are asked to 
record and embed a simple wave file into the Wiki, convert a Wav to Mp3 via a “free” online media 
converter, and create a simple Google presentation to embed. During subsequent modules as students’ 
efficacy climbs they are directed toward more complex technological developments such as multi-tracked 
audio and video outputs using programs that balance user friendliness, effectiveness, and relatively free 
availability. These are characteristics that are likely to encourage pre-service teachers to continue using 
technologies adequately when they transition to in-service status; a time that has many reeling from the steep 
learning and time commitment curve common during the first two years. Toward the end of the term students 
in the Technology for Educators course are pushing some of the low-end technologies to their limit, which 
inevitably impacts their output and perception of technology. 
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3.8 Student Media Preference 


When students were asked if they preferred demonstrating content they’ve learned in the course through 
audio or video output (reading responses, interactive hypermedia, videos, etc.) over writing a paper on the 
same 81% strongly agreed, and 19% agreed. Given the number of technical glitches that were communicated 
during the term it is curious that no students indicated preference for writing a pontification paper over 
creating the video. Perhaps as Roblyer and Doering (2010) point out technology can improve student 
motivation, attitude, and interest in learning. In an end of course improvement evaluation the instructor 
administers to students a rank-order question indicated that students prefer audio and video enriched 
technologies (see Fig. 1). The question only analyzed the larger project outputs without sub-analysis of the 
smaller technologies that most often fed into the larger projects. In the ranks for each project/media type 
students identified Podcasting/ Audacity (88) as their optimum output medium with E-Learning/PPT (81) and 
Video Pontification/Movie Maker (77) close behind in that order. In discussions between the instructor and 
students most seemed more enthusiastic about the outcome of their E-Learning and video activities, but the 
researcher believes the higher rate of technology glitches and higher complexity level of the assignments 
associated with their development led to lower ranking than the Podcast. Large file sizes, program freeze-ups, 
conversion to an iTunesU compatible mp4 file format, and student self-consciousness about presenting in the 
video were common concerns voiced by students during the latter part of the term devoted to the multimedia 
projects. 
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Figure 1. Class rank-order preference of major projects 


3.9 Student TPACK Perception 

Students (N=21) were given an additional questionnaire, the Technological Pedagogical Content Knowledge 
Alignment Perception Scale (TPACKAPS) at the beginning of the semester, and again at the end. Each 
student was asked to rate various components of the course (readings, discussions, papers, and media) on a 
scale from 1 to 10 for technology, pedagogy, and content knowledge emphasis (1 = none; 10 = primarily). 
Paired-samples t tests were conducted to determine whether students’ perception of the course components 
varied upon completion of the course. The results indicated that the pre and post means (Table 1) varied 
significantly at the p < .01 level in students’ perceptions of videos and podcasts for technology, pedagogy, 
and content knowledge. The positive correlation indicates that students perceive more technology focus in 
the beginning, but they perceive more pedagogy and content knowledge in the media projects. When asked if 
they felt that media output represented and demonstrated their level of learning in the course with regards to 
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technology, pedagogy, and content knowledge 76% strongly agreed, 14% agreed, and 10% neither agreed nor 
disagreed. 


Table 1. Student TPACK perception means 



Video 

Podcast 

Pre 

Post 

Pre 

Post 

Technology 

8.67 

7.81 

8.86 

7.62 

Pedagogy 

3.95 

7.91 

3.95 

8.24 

Content Knowledge 

4.86 

8.48 

5.14 

8.10 


4. EDUCATIONAL SIGNIFICANCE 

The results of this study indicate that students prefer multimedia over other types of technology, view digital 
video and audio as TPACK rich media capable of demonstrating their competencies. Given that the course 
utilized in this study is heavily infused with large doses of pedagogy and content knowledge instruction in 
addition to the technology literacy skill development a balanced TPACK approach is modeled for the 
students. Furthermore, students are challenged to create outputs that equitably merge each TPACK 
component. 

Aside from an instructional and learning tool video has been around for many years as a formative 
assessment, feedback, and planning tool. Common uses in this realm have included recording oneself giving 
a speech or presenting a student teaching lesson, real-time and post-game sports analysis, diagnosis of 
medical conditions or behaviors, pre and posttest analysis of research subjects, and anything that requires a 
comparative stop-action, archival capability. From an instructor’s point of view the digital audio and video 
output can offer a creative and visual dimension not offered in print. From the student perspective whereby 
they are interjecting an audible or visual presence in the media it is typically more customary as a self- 
assessment tool. Video tools are not uncommon as a means to teach content to others, even with the self in 
the visual mix. Through the use of audio and video students are able to solidify their learning due to the 
increased cognitive processing needed to develop quality output. In addition students will encounter added 
motivation due to prospect of having a novel means to demonstrate their competency. As a demonstration of 
what one has learned based upon engagement in a course, however, digital audio and video in the online class 
environment is an under-utilized and viable output. It warrants further analysis as an output tool not only in 
Educational Technology, but in less technology-focused disciplines as well. 
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ABSTRACT 

The diffusion of the use of the learning virtual environments presents a great potential for the development of an 
application which meet the necessities in the education area. In view of the importance of a more dynamic application 
and that can adapt itself continuously to the students’ necessities, the “U-ALS” (Ubiquitous Adapted Learning System) 
was proposed and developed. This system was built based on the learning virtual environment Moodle and on the Module 
Mle-Moodle, available in a Cloud Computing infrastructure and it has as a main purpose the adaption to the student’s 
computational context, glimpsing technical characteristics like the environment adequacy to the user’s speed connection. 
The obtained results show the feasibility of working with systems that are sensitive to the context, bringing 
improvements in the students’ access to the materials and tools. 
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1. INTRODUCTION 

The use of the Virtual Learning Environments (VLEs) in the academic environment has become very 
common. The use of these environments with the support and help in the teaching and learning processes 
make the VLEs an important pedagogical resource (RIBAS, 2009). In this case, making the environment 
adequate to the student, considering its individual characteristics is a necessity today. In this field the 
ubiquitous learning (U-learning) is inserted, which enables the access to the educational resources with total 
mobility and adaptation of the system to the students' computational context. In this context, this paper seeks 
for proposing a new application for these environments where the students, besides having the facilities of 
mobility and adaptation of the device through the Mle-Moodle, they will have an environment which will 
adjust itself to its computational context, making the materials adequate to the different speed connection, 
available in a Computational Cloud infrastructure which guarantees availability and access through different 
devices. 

The Cloud Computing emerges as a technology to improve and make the use of computational resources 
more efficient, through characteristics like availability, elasticity and the adaptability of services, where the 
user will be able to access his programs remotely and he can have a larger capacity of storage and processing 
available, without the necessity of owning more expensive equipments, because his data will be available in 
the cloud. In this computation model, the technology resources are allocated on a scalable way, this way 
these services are offered to external clients through technologies via Internet (CEARLEY, 2010). 

According to David Cearley, the clients don't need to know how it works; they will simply be able to use 
the offered services. Thus, Cloud Computing is a model in which the computation (processing, storage and 
software) is somewhere in the net and it’s accessed on a remote way (CEARLEY, 2010). 

The increasing of size, complexity, the number of users in the internet problem, as well as technological 
diversity that the same use to access the net brings the necessity of treating each user on a specialized and 
individual way, identifying the profile and developing applications in virtual environments able to adapt 
themselves. 

The existent VLEs own static platforms, enabling only the selection of the contents and the tolls that will 
be used by the teachers, without considering the students’ individual characteristics that will use the 
environment. 
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The trend to the VLEs’ evolution is to evolve to aggregation tools which seem to be able to help to 
materialize the PLE (Personal Learning Environment) concept (OLIVER AND LIBER, 2011). The central 
idea of PLE is to give the student or, more genetically, to the learner, a personal space where it adds the 
contents that are useful, produced by others, but also the ones that it produces itself, and that this way 
provides according to its interest. In other words, instead of using a designed space similarly to a group of 
students, as it happens in the traditional VLEs, it creates its own space as well it keeps it under control. This 
evolution has been referenced as the transition from a “one for all” system to “one for me” systems and it 
permits to highlight the control question, being the learner able to determine his own choices and action 
course (HARMELEN, 2006). 

From the researches done through a speed measurer integrated to the VLE Moodle and to the Module 
Mle-Moodle of the X School (XX), we noticed through the register of accesses of the environment for 
desktops and of the accesses in the environment for mobile devices, the several ways that the students can 
access the environment, through mobile devices or desktop computers and the great range of connection 
speed, showing that the students’ computational context bring a problem in the usage of the VLEs, because 
with lower speeds of connection problems of access to bigger files arise, because it takes them a long time to 
be loaded, transferred or open and to use tools that need a larger broadband internet connection like the chats. 

This work makes the VLE Moodle dynamic, with characteristics of a software u-learning through the 
creation of a module that can identify the student's computational context when he enters into the course. The 
module identifies the student's connection speed and adapts the contents and the tools, that with the module 
Mle-Moodle brings to the environment the characteristics of availability, mobility and sensibility to the 
context. With this work, the VLE Moodle and the Module Mle-Moodle became not static and didn't serve as 
simple repository of information where the teachers put their subjects and they can just select the tools that 
they will use to become an environment that identifies user’s variables that influence in the way that the 
teachers access the environment and in its usage. The Science Computer usage in the Clouds brings the 
advantage that it’s not necessary to keep the hardware infrastructure, enabling the increase of the storage 
capacity, memory and processing because of the increase in the number of students, dispensing the 
maintenance of the TI staff to keep, update and reconfigure the environment according to the necessities. In 
the education area the Science Computer technology usage in the Clouds justifies itself by the fact that the 
informations can be accessed from anywhere, through the Internet, the elasticity of available resources and 
the wide access through personal computers, smart-phones or PDAs. 

For better comprehension of the research and the results found, this article is structured in six chapters, 
being the second chapter the presentation of some concepts referent to the cloud computing; in the third 
chapter are presented definitions about mobile VLEs; in the fourth chapter are presented some correlated 
works; in the fifth chapter is presented the methodology and the development of the Moodle U-ALS; in the 
sixth chapter are presented the results and discussions. 


2. CLOUD COMPUTING 

Cloud Computing is a new trend of technology that intends to be global and to provide services to people that 
go from the final user that hosts personal documents in the Internet to companies that outsource all the TI 
infrastructure (Technology of Information) (BUYYA el al, 2009). The Cloud Computing is justified by the 
fact that as the software resources as the hardware ones, get obsolete in a short period of time, making the 
computing platforms usage of third ones the ideal solution to the infrastructure problems. To use their 
systems the user doesn’t need high computing resources, being of software or hardware, decreasing the 
machine purchase cost (MACHADO and MOREIRA, 2010). 

The infrastructure of the Cloud Computing environment is usually formed by several physical machines 
connected by a net. Each machine has the same configuration of software, but they can range the hardware 
capacity related to the processing, storage and memory power. Inside each physical machine there are a 
variable number of virtual machines or nodes in execution, according to the physical machine (SOROR et al, 
2010 ). 

Nowadays there are several technologies that permit the creation of a new infrastructure of cloud 
computing, among them: Amazon Web Service, Google App Engine, OpenNebula, Nimbus e a plataforma 
Eucalyptus, chosen for this work. 
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2.1 Eucalyptus 

It’s an open code infrastructure that permits the creation of a compatible infrastructure for the users can try 
the Cloud Computing (LIU, LIANG and BROOKS, 2007). The architecture of the eucalyptus is simple, 
flexible and modular and it also has a hierarchical conception that shows the common resources of the 
environment. 

An installation in Eucalyptus cloud can add and manage resources from only one cluster or several ones. 
The cluster is a group of machines connected to the same LAN. It’s possible to have one or several instances 
of a NC (Node Contrller) in a cluster, and every one of them manages the instantiation and the termination of 
virtual instances (IBM, 2011). 

According to Machado (2010), the Eucalyptus has as a purpose to help the research and the development 
of technologies for Cloud Computing, with simple implementation using tools for administration and help to 
the users’ and system management, with the ability of setting several clusters. 


3. VIRTUAL ENVIRONMENT AND MOBILE LEARNING 

In the last decades, we have been experienced the appearance of a connected mobile society, with a variety of 
information resources, technologies and ways of communication available. Before the advancing that the 
Mobile Computing is achieving, is the fact that, the usage of mobile devices is reaching all kinds of users and 
usages, among them the usage in VLEs (MOZZAQUATRO, 2010). 

The Moodle is a software of orientation Open Source (free), its development objects the management of 
learning and collaborative work in virtual environment permitting the creation and the management of online 
courses, work groups and learning communities. In Brazil, the Moodle was approved by MEC as platform for 
Distance Education, which will be able to be adopted by any institutions that want to apply this kind of 
teaching. The Moodle presents several resources, like discussion forum, diaries, glossaries, tasks, chats, 
questionnaires that can be selected by the teacher/manager in a way to create a more flexible learning 
environment, that attend to the pedagogical goals and to the students’ necessities, providing a module to the 
access via mobile device too (FRANCO, 2010). 

The possibility of personalizing the environment where the student interacts in the teaching search, 
helping all the ones involved in the teaching/learning process to get their goals, it makes the VLE dynamic. 
An important factor during the student's accompaniment process is to be aware about the context where the 
student is acting, enabling the adaptation of teaching strategies according to the specific student’s reality, 
enabling an adaptive support, where each student will have their characteristics referent to the environment 
and to the presented contents. This adaptation of the environment is characteristic of the Sensitive 
Environments to the Contexts that are presented. 

3.1 Environments Sensitive to the Context 

Sensibility to the Context refers to all that happens around the user, and that influences the way as he 
interacts with the environment and with the other people. To the ubiquitous computing, sensibility to the 
context is similar, where computing systems can perceive the context and interact according to him 
(PERNAS et al, 2009). 

Sensibility to the context refers to the capacity of a class of systems to the usage of the contextual 
information to offer of better services to the user, in a flexible and manageable way (DEY et al, 1999). A 
system that is sensitive to the context is able to extract, interpret and use the contextual informations and to 
adapt its functionality to the current context of usage, to provide services to a person particularly, place, time, 
event, etc (BYUN and CHEVERST, 2004). 

Each student accesses the VLE under determined conditions. The student can be at home or in another 
environment, he can use a personal computer or a mobile device, he can have knowledge acquired from other 
sources or having no knowledge about the content, he can have done part of the activities or done none of 
them (PERNAS et al, 2009). So, each student has his situation characterized and too individual, so to 
determine these variables that can interfere directly in the student’s teaching is a very important task. 
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3.2 U-Learning Environments 

To Chiu (2008), u-learning is a system aware of the context and that can fell the student’s informations and 
the informations around the students in the real world, and then can offer personalized services. So, the 
students can learn the knowledge, abilities and problems and be able to solve while interacts with the real 
world through authentic scenarios. 

To Weiser (1993), the computers should be embedded on a implicit way to the user’s environment, with 
human-computer interaction on a non-intrusive way, without the artifacts usage imposition like keywords and 
remote-controls, being this one closer to the way that the human-been gesticulate, talk or write to 
communicate. 

U-leaming environment, in counterpoint to the traditional VLEs that usually are static and own contents, 
structures and static presentations, they object to provide an environment with functionalities that permit the 
adaptation of this environment to the specific situation lived by the user in every interval of time, thus, a very 
innovative propose. 

In the educational area, the focus of this work, many virtual environments of learning are not more than a 
static repository of content, with the same materials, structures and presentation to all the students 
(GASPARINI et al, 2009). 


4. WORKS CO-REPORTS 

Since 1994 we notice the initiatives of software development with context treatment, we can mention: the 
Shopping Assistant (1944), the XeroxParc (1996), the Cyber guide (1996), the Conference Assistant (1999), 
the Lookout (1999) and the Campus Aware (2002). 

The SEDECA objectified to identify and to adapt the VLE MLE Moodle to the student's cognitive style 
through a system created to diagnosticate learning styles. The analysis of these categories permitted to define 
indicators that enabled the adaptation of the learning virtual environment MLE Moodle to these different 
cognitive styles to the adaptive hypermedia. The materials and proposed activities in the adapted 
environment were presented according to the four cognitive styles that highlighted the most in the research: 
Holist, Serious, Divergent and Reflexive (MOZZAQUATRO, 2010). 

The Adapt Web (Adaptive Teaching-Learning Environment on the Web) is an AS (Adaptive System) of 
EAD based on the web, open source. The Adapt Web has the purpose of adapting the content, the 
presentation and the navigation according to the user’s profile. Its adaptation is supported by the creation of a 
student’s flexible model (user’s model), where, for each student are saved personal informations like his 
background, knowledge, preferences, navigational historic and technological resources (PALAZZO et al, 

2008) . The educational contents are organized by a hierarchical structure of concepts, establishing criterions 
of pre-requirement. The structure is defined during the authorship level and it is stored in the XML format 
(Extensible Markup Language). These XML documents pass by an adaptation before being presented to the 
students. The adaptation occurs as in the content, as on the interface and in the navigation (PERNAS et al, 

2009) . 


5. METHODOLOGY AND DEVELOPMENT OF THE U-ALS MOODLE 

Talking about the nature of the research, this paper presents an applied research, because seeks for the 
solution of a specific problem and about the technical procedures used, the research is classified as 
experimental research, because it selects variables that are able to influence the studied goal (SEVERINO, 
2000 ). 

The general goal of the U-ALS Moodle development was to provide, in a Cloud Computing 
infrastructure, an adaptive environment as to desktop computers as to mobile devices providing an 
environment adequate to the student's computing context, through the adaptation of the content and of the 
available interface, according to the speed connection. 

The proposed architecture presents an organization in two distinct layers, the Interface Layer and the 
Communication Layer, more the integration with the DBMS (Data Bank Manager System). On the Interface 
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Layer, where the student accesses the content to do the activities, the content will be already available 
according to the each student’s speed connection. On this layer the contents and the tools are accessed. On 
the Communication Layer, the architecture proposes an Adaptation Agent that has the function of treating the 
executed informations on the Interface Layer by the students, providing the adequate material. 

The treated informations are the speed connection identification, which is stored in the XML file 
(Extensible Markup Language). From this the Adaptation Agent modifies the environment for the student to 
have an adequate navigation. The adaptation of the environment is done in the provided materials and also in 
the tolls that are provided according to the student’s net context, being the variable analyzed by the Agent 
and the access speed Variable to the VLE. The Agent is going to communicate with the Data Bank Manager 
System (DBMS) and to adequate the environment to each student. 

Despite the DBMS saves the inserted informations, the stored variables to the adequacy of the 
environment to the student’s profile, considering its speed as IP, connection speed and student’s access date 
in the VLE are stored in an XML file, which is saved inside the Moodledata, placed in the Cloud provider 
created, and these data are useful for posterior statistic analysis. The student's speed identification will 
indicate what contents and what tools will be provided in his access to the environment. 

The Agent that does the content adaptation watches the environment while the student is logged-in, at 
each interval of 60 seconds showing the speed on the screen and saving the data. If the student has his speed 
reduced, the environment adapts itself to this speed and if the speed increases the Agent also provides 
materials and tools to this connection. 

The material to be used by the students is stored in distinct categories, one for low speeds and other for 
normal and high speeds, making the environment totally personalized, and that the student will only visualize 
the ideal files for his connection speed, as well as the adequate tools to his navigation. 

The cloud infrastructure provides to the development of the U-ALS, is based on nine computers, with the 
following format: a gateway that permits the access to the environment through the net; a provider with the 
cloud controller, cluster controller and storage controller functions and; seven machines serving the node 
controller, which ones provide the cloud processing. All the machines used in this environment, as node 
controllers, own the technology of native visualization in hardware, being six computers powered with Dual 
Core processors and one with Quad Core Processor, totalizing the maximum of sixteen cores in the 
computing cloud. 

The Eucalyptus framework choice was because it is open source and to enable the implementation of 
scalable and efficient clouds. Through the virtualization it is possible that several instances of the operational 
system be executed simultaneously in only one computer, being possible to control the usage of the CPU, 
memory and storage and even to permit that the operational system migrates from a machine to another. In 
this paper, the KVM (Kernel -based Virtual Machine) was used to the virtualization. The KVM is native in 
the Ubuntu Operational System, being possible to load Windows and Linux clients being used to raise the 
instances in the Eucalyptus framework. 

The Cloud Computing usage brings the elasticity of resources advantage to VLE, because if the number 
of users or the necessity of more space to store more content, it is not necessary the purchase of more 
hardware resources like new providers with more space or more power of processing, the necessary resources 
are automatically allocated by the cloud, and the reallocation of resources can be done in real time, without 
the necessity of stopping in the system. 

The computing cloud implantation model is private, and the service model is a kind of SaaS that has as a 
purpose to make easier and more accessible the resources providing to the students. 

The U-ALS works in a transparent way to the user, when the student enters into the environment, this one 
does not present any apparent change, having characteristics of U-leaming software. When the student clicks 
on the course the system will check its connection speed and it provides the materials that are adequate to 
that connection usage. The U-ALS also provides the tools that are showed. 

The adapted environment stores the students’ connection speeds and the adapted materials that were 
accessed, making the students' profile study possible, with the possibility of indicating improvements of the 
VLE conditions to attend the students’ necessities, and thus influencing in the final quality of the course. 
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6. RESULTS AND DISCUSSIONS 

The validation of the U-ALS Moodle was done at X (XX) School, in the subjects about Wireless Technology 
and Computer Architecture, where were collected data that will be presented in this paper. The Figure 1 
presents the U-ALS Moodle screen showing the users’ connection speed and the adapted materials, in the 
Wireless Technology Subject. 



Figure 1. U-ALS Moodle with Adapted Materials 

The graphic of the Speed Range, of the Figure 2 in the horizontal position, “X” axis, presents the five 
lanes of speed divided to present the students' range of speed who accessed the environment and the vertical 
axis, “Y” axis, presents the number of students that accessed the environment, making this way the 
connection among the number of students with the access speed. We can notice that most of the students still 
have connection speed considered low comparing to the available speeds, until 15 mega via broadband 
connection (OI, 2011). 



Figure 2. Graphic of Speed Range 

The graphic shows the importance of the creation and availability of environments sensitive to the context 
in the computers net area to an improvement in the quality of the students’ access to the materials. The static 
VLE, without identifying the student’s speed variables indicate that about 70 accesses of the 200 selected 
accesses in the research would show problems, in other words, 36% of the students would have problems 
watching videos or opening heavier files. If adapted materials were not provided these students would end up 
without the contents or partially presented and non-stimulated to access the environment again. 

The Figure 3 presents the Graphic with the number of students that accessed adapted contents in the 
period. The Graphic shows that the selected sample, 64% of the students that accessed the environment did 
not need that the materials were adapted, but 36% of the accesses presented a computing context that needed 
that the tools and the materials were adequate to a better AVA usage. From the 200 selected accesses, 72 of 
them were with connections with until 500 Kbps as we found in the Graphic of the Picture 2, of the Speed 
Range, with this, 36% of the students had access to the differentiated materials, showing this way the 
necessity of each time that the VLEs treat each student in a personalized and individual way. 
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i Alunos que Acessaram Materials Adaptados 
Alunos que Acessaram Materials Normais 




Figure 3. Graphic showing the number of students that accessed adapted materials. 

One of the problems of the usage of a standard virtual environment (unique for all the students) and that 
is about different user’s net contexts is in the fact that many limitations in the access occur, decreasing the 
environment usage by the students that have speed limitations. This can reflect in the resistance or even in the 
abandonment of using the environment, if a course in the distance modality, can also influence in the course 
evasion. 

It is necessary that all the VLE structure help in an effective way on its usage, being necessary that 
alternative ways of access be available, like the use of mobile equipments and also that the environment have 
guarantee of availability, what in this U-ALS Moodle is granted through the created cloud. 

From the finding that the same user can access the environment in more than one way, using different 
equipments like computers, laptops, smart-phones or tablets and from different places, the student can access 
from home or from school, it becomes extremely necessary the usage of technologies like Cloud Computing, 
because this guarantees that the environment will be always available when the student chooses to do his 
works. We also found that despite most of the accesses occur in the morning and in the afternoon; some 
students accessed the environment during the night, increasing thus the necessity that the environment be 
always available. 

The elasticity of resources is also another advantage found from Computing Technology in Cloud, 
because the teachers have, thus, the possibility of creating new courses and register new students without 
worrying about the physical part of the structure where the VLE is installed. The facility of adding resources 
is another advantage found, because the teachers don’t need a technical staff to maintain the system working, 
not having investments with the TI staff and with the infrastructure. 

This new scenery shows the contributions of the U-leaming environment to the course development as on 
the presence modality as on distance, making necessary the implementation of systems that make the access 
to the easy and stimulant to the student. It is necessary that all the structure of these environments be 
adequate to the student and work in a way that provides diversity and facility of access, that’s why the 
necessity that the virtual environments be improved and adapted arises more and more to their users’ 
individual characteristics. 
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ABSTRACT 

This research follows on previous studies that investigated how digitally augmented devices and knowledge scaffolds 
enhance learning in a science museum. We investigated what combination of scaffolds could be used in conjunction with 
the unique characteristics of informal participation to increase conceptual and cognitive outcomes. 307 students from nine 
middle schools participated in the study. Six scaffolds were used in varying combinations. The first was the digital 
augmentation. The next five were adaptations of knowledge-building scaffolds. Results demonstrated greater cognitive 
abilities in terms of theorizing can be obtained with particular combinations of scaffolds, however, there appears to be a 
trade-off in what students can learn conceptually in informal environments in which learning takes place in short 
episodes. 


KEYWORDS 
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1. INTRODUCTION 

This research seeks to understand how aspects of knowledge building pedagogy (Scardamalia, 2002; 
Scardamalia & Bereiter, 2006), known to be successful in formal classrooms, might be adapted to reach 
broader informal audiences to improve scientific literacy in museums. The recent National Research Council 
(NRC) report on learning science in informal environments (NRC, 2009) highlights the potential of non- 
school settings, such as museums, for engaging large populations in scientific investigation. Despite this 
potential, the NRC report outlines the need for essential research in three key areas. First, while there is 
ample evidence that suggests informal environments increase engagement and interest, fewer studies have 
focused on how those experiences result in conceptual gains of science content. Second, in terms of scientific 
skills, designed interactives have been shown to increase lower level skills such as manipulating and 
observing, however more challenging skills such as critical thinking and theorizing are less frequently 
demonstrated. Finally, as digital platforms are increasingly incorporated in informal settings, more research is 
needed to determine how they enhance the learning experience. 

The research reported here follows on a series of studies that investigate how digitally augmented devices 
and knowledge-building activities can enhance learning in a science museum (Yoon et al., in press-a; in 
press-b). In the series we use a quasi-experimental design in which students in multiple conditions interact 
with a museum device using digitally augmented information and varying arrangements of knowledge 
scaffolds. Results thus far have suggested that digital augmentations can help in conceptual development and 
that students’ abilities to interpret information about a phenomenon using knowledge-building scaffolds can 
improve cognitive understanding in some conditions. However, one major concern that has emerged in our 
observations of student interaction in the more highly scaffolded condition is the over-formalization of the 
learning experience that appeared to be needed to achieve those results. Specifically, when students were 
asked to use learning scaffolds that were delivered in, for example, worksheets with knowledge prompts, 
students were less explorative and referred to the scaffolds to dictate their next steps. Students in less 
formally structured conditions were generally more playful and experimented on their own — much like 


205 



ISBN: 978-989-8533-12-8 © 2012 IADIS 


students would normally behave in museum exhibits during a school field trip. Informal learning 
environments can enhance interest and engagement because the experience does not look like school, i.e., it 
is fluid, sporadic, and participant-driven (Falk & Dierking, 1992, 2000). The research in this study aims to 
investigate the issue of over-formalization. Specifically, the research question we seek to answer is: Which 
combination of learning scaffolds can optimally be used in a science museum taking into consideration the 
unique characteristics of informal participation to increase learning outcomes? 


2. BODY OF PAPER 

2.1 Theoretical Considerations 

2.1.1 Augmented Reality as a Scaffold for Learning 

In the most recent Horizon Report , the New Media Consortium (2012) discusses the enormous potential 
augmented reality (AR) capabilities have on learning and assessment in enabling people to construct new 
understanding. AR experiences layer digital displays over 3D real world environments (New Media 
Consortium, 2012) providing access to normally hidden data that users can use to develop deeper knowledge 
about a phenomenon or take a different immersive perspective to broaden their understanding. In the past 
decade, practical uses of AR have emerged in fields such as games, marketing and advertising, films, 
navigation, and for medical and military applications (El Sayad et al., 2011). Although newer in education, 
over the last few years, there have been studies that illustrate AR’s potential for learning particularly in the 
field of science education. For example, Dunleavy and colleagues (2009) document high student engagement 
influenced by the ability to collaboratively problem solve and collect data in the real world in their Alien 
Contact! handheld AR environment. Squire and Klopfer (2007) detail the impact of their AR game 
Environmental Detectives on accessing student prior knowledge by connecting academic content to physical 
spaces that students are familiar with. In Outbreak @ The Institute, Rosenbaum and colleagues (2007) 
document the affordance of their augmented reality game play to include authentic scientific inquiry and 
understanding the dynamic nature of system interactions. In these studies, the indirect correlates of student 
learning, i.e., engagement, prior knowledge, and processes in scientific practice, are important outcomes of 
the research and provide valuable impetus for pursuing further studies on what and how students learn in 
terms of scientific knowledge. 

AR technologies have also been incorporated in museums to enhance visitors’ experience by improving 
their interest, engagement, and access to information (Baber et al., 2001; Damala et al., 2008; Hall & 
Bannon, 2006). For example, Szymanksi and colleagues (2008) examined how two different electronic 
guidebook prototypes affected visitors’ social interaction. As visitors toured a historic house, the guidebooks 
provided information about the historic artifacts that the visitors encountered. Results indicated that these 
guidebooks led visitors to engage in more content-rich discussions with each other. Furthermore, their 
exploration of the room and its objects were enhanced (Szymanski et al., 2008). Similarly, Baber et al. 
(2008), in testing three different AR platforms in an art museum, discovered that different devices elicited 
varying lengths of visitor viewing times and visitor preferences. Damala et al. (2008) also tested an AR- 
enabled mobile multimedia museum guide in a fine arts museum and found that visitors enjoyed the playful 
content presentation that the museum guide enabled. Finally, Hall and Bannon (2006) investigated the 
effects of a digitally augmented exhibit on children. When children were given RFID sensors that could 
detect exhibit locations and unlock virtual information, their interest and engagement increased. 
Collectively, these studies demonstrate the impact that AR can have on visitors in a museum setting in terms 
of increasing engagement, interest, and access to information through scaffolded experiences. 

Encouraged by this previous research and in envisioning potential extended uses of AR in museums as 
scaffolds for knowledge improvement, the present study addresses the real need for learning research (NRC, 
2009) in informal environments through connecting with the notion of scaffolds. In the next section, we 
describe our rationale underpinning the use of scaffolds and in particular the use of knowledge-building 
scaffolds to improve learning. 
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2.1.2 Learning through Knowledge-Building Scaffolds 

The use of scaffolds in educational technology applications has been researched fairly extensively to support 
scientific inquiry and cognitive tasks (e.g., Quintana, 2004). In particular, a long-standing program of 
research in the learning sciences that is premised on designing learning environments through the intentional 
application of technological and pedagogical scaffolds is knowledge building (Yoon, 2008; Bereiter, 2002; 
Scardamalia, 2002; Scardamalia & Bereiter, 2006). This approach is centrally focused on the goal of 
improving ideas in the same way that knowledge work is done by experts in real world contexts (Scardamalia 
& Bereiter, 2006). Primarily applied in school classrooms, knowledge-building studies have been shown to 
increase student scientific abilities in explanation, interpreting and evaluating information, and knowledge 
advancement (van Aalst, 2009). Students also acquire deep theoretical understanding of scientific 
phenomenon through collective sustained inquiry and research on problems that can range from what causes 
leaves to change color in the fall in a grade 1 classroom (Scardamalia, 2002) to the complex influences on 
genetic engineering research with middle and high school students (Yoon, 2008; 201 1). 

The technological application. Knowledge Forum and associated pedagogy use educational scaffolds to 
enable public, collective contributions that shape the knowledge constructed in the learning community. Such 
scaffolds include prompts for consensus building, generalizations, differentiation between evidence and 
theories, and peer evaluation. For example, a prompt such as “My theory is...” encourages students to use 
evidence to construct a more general understanding of a class of scientific phenomena. Similarly, students 
can create a “rise above” note, enabled by the archived database of peer exchanges, which is a distillation of 
an idea or theory from a collection of previous peer exchanges that provide students with opportunities to 
think across diverse perspectives and to arrive at conclusions about how the collective learning community 
views a scientific issue (Yoon, 2008). 

Collaboration also factors prominently into the knowledge-building approach. By working with others 
discursively to problem solve, evaluate evidence, and identify important shared understanding , students are 
able to more deeply reflect on what they know rather than learning independently, or learning through textual 
modes. This decentralized, public, and distributed participation promotes what Scardamalia (2002) calls 
collective cognitive responsibility where the impetus for learning is generated by consensus within the 
community rather than by the teacher. From this set of theoretical and pedagogical descriptions, our series of 
studies uses varying degrees of what we collectively refer to as knowledge-building scaffolds which include: 
knowledge prompts, a bank of peer ideas, working in collaborative groups, instructions for generating 
consensus, and worksheets for recording shared understanding. However, because knowledge building 
requires the development of a community with shared understanding, language, and goals, learning events 
evolve over longer periods of time than informal environments may afford. Van Aalst (2009) characterizes 
learning experiences that are less focused on the community as knowledge construction in which students 
may collaborate in small groups on tasks that require less synthesis and reflection on the knowledge 
advancement process. We have understood the limitation of our informal setting and population in terms of 
achieving a true knowledge building community in previous studies (e.g.. Author, in press-b) but have 
nevertheless attempted to investigate how aspects of knowledge building pedagogy can be applied in 
informal environments given its success in formal classrooms. The issue of over-formalizing the experience 
was a phenomenon that emerged as an important context-dependent factor in considering how informal 
learning takes place (Author, in press-b), which is the subject of this study. In the next section, we describe 
the features of informal environments that make learning unique. 

2.1.3 Features of Informal Learning Environments 

Learning in informal spaces is fluid, sporadic, social, and participant driven — characteristics that contrast the 
highly structured formal classroom experience (Honey & Hilton, 2011, NRC, 2009, Squire & Patterson, 
2009). Activities are often experienced in single -visit episodes (Falk et al., 2007) where visitors learn on their 
own with little structured follow-up or reflection. McManus (1994) has characterized typical visitors as 
demonstrating scouting behaviors within museum exhibits, where they roam around, encounter devices, and 
act quickly to discover the intended information. Thus, more systematic learning studies are difficult to 
design. However, science museum exhibit developers do intentionally design learning spaces that mix a 
variety of supports for learning. For example, exhibit devices (individual interactive kiosks within the larger 
exhibit) are deliberately grouped and arranged to encourage progressive engagement with topics (Grinell, 
2006) in support of the exhibit’s overall learning objective. Many devices are accompanied by posted graphic 
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panels (Serrell, 1998, 2006) that provide printed content in order to support the interpretation of scientific 
phenomena. Finally, visitors often encounter exhibit educators (Serrell, 2006) who are skilled at facilitating 
learning in social groups around exhibit devices. They spark conversation, notice ideas, and encourage 
theorizing. We see parallels between these aspects of the informal learning environment and the structure of 
effective knowledge building in classroom settings. 

For this study, we deliberately attempted to construct conditions that probe these parallels. The 
application of a digital augmentation to an exhibit device provides a first layer of interpretive support and 
acts as a primary scaffold. We then added posters and worksheets with progressively more rigid layers of 
structure around the experience to advance the scaffolding for analysis. In some conditions, we prompted or 
required group work as a strategy for measuring the benefit of collaboration. We attempted to detect the 
frequency of theory building and, in some conditions, prompted it explicitly. As our methods and results will 
show, these scaffolds had varying levels of success. 

2.2 Methods 

2.2.1 Participants and Context 

For our study population, we recruited teachers to bring their students who had previously participated in 
workshops and other teacher events at the museum or were referred to us to participate in the study by those 
teachers. In total, 307 students (52% female, 48% male) on a school fieldtrip from nine local middle schools 
(grades 6-8) participated in the study. We chose the middle grades for this study to have confidence that the 
students were developmentally able to theorize. Also, the study engaged students with the topic of electrical 
conductivity. By grade six, all students would have encountered the topic in their classrooms, as dictated by 
the local standardized curriculum. This provided a common ground for the participants, which increased the 
chances that they would have similar prior knowledge. 

An exhibit device called “Be the Path” provided the context for the study (see Figure 1). The device 
consisted of two metal spheres on a table, approximately one foot apart, with one connected by a wire to a 
battery and the other connected to a light bulb. The instructions on the device provided little direction, simply 
suggesting “try to complete the circuit.” The students attempted different configurations to complete the 
circuit and light the bulb. Once the circuit was completed, a projected visualization of electrons flowing 
around the complete loop appeared. A more complete description of the technological platform and how we 
developed it can be found in (Author, in press-b). 



Figure 1. “Be the Path” device with digital augmentation 


Students were randomly assigned to conditions, constructed to represent increasing use of scaffolds. Six 
scaffolds were used in varying combinations. The first was the digital augmentation. The next five were 
adaptations of knowledge-building scaffolds which included: 1) group work; 2) worksheet questions 
directing student attention to relevant information; 3) directions about how to work together; 4) embedded 
worksheet knowledge prompts; and 5) bank of previous worksheet responses. Condition 1 (Cl) served as the 
control group with no scaffolds. Condition 2 (C2) represented the digitally-augmented device with no 
additional scaffolds. Condition 3 (C3) was the same as C2 but with posted worksheet questions such as What 
happened when you touched both metal spheres?'. What happened to make the bulb light up? Students in Cl- 
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C3 used the device individually and completed individual worksheets afterwards. Condition 4 (C4) students 
worked in groups, with posted worksheet questions, completing the worksheet individually after the 
experience. C4 was meant to represent the typical informal learning state in which groups of students 
encounter posted labels with questions about devices. Condition 5 (C5) students had the same treatment as 
C4 students with additional knowledge prompts. Knowledge prompts included. Our hypothesis is; Our 
Theory is; and Others have said. A bank of other student worksheet responses was also posted for students to 
refer to as they answered the worksheet questions. They completed the worksheet collaboratively after the 
experience with one worksheet per group. Condition 6 (C6) students had the same treatment as C5 except the 
worksheet was completed collectively during the experience. Table 1 shows the number of students who 
participated in each condition. 

Table 1. Number of students who participated in each condition 


Condition 

Number of Students 

1 

30 

2 

31 

3 

31 

4 

24 groups of 3 (72) 

5 

23 groups of 3, 1 group of 2 (71) 

6 

24 groups of 3 (72) 


2.2.2 Data Sources and Analysis 

Three data sources were collected and analyzed through a quasi-experimental mixed-methods approach: 
surveys, worksheets, and interviews. Each data source is detailed below. Surveys ; A conceptual knowledge 
survey was administered to students in each group before and after the intervention. The survey posed five 
general multiple-choice content questions each valued at one point, related to the scientific topic of electrical 
conductivity and circuits. An additional open-ended question on the survey also solicited responses that 
demonstrated knowledge directly related to the device experience, i.e., “Think about an electric circuit that 
supplies electricity to a light bulb. What parts make it work so that the bulb lights up?” Responses to this 
question were coded on a five -point Likert-scale from no understanding (0) to complete understanding (4). 
For the coding scheme, the dataset was reviewed qualitatively by members of the research team in a series of 
data analysis sessions during which responses were compared to answers in a teacher’s manual. We also 
consulted a physicist on staff at the museum to help the team design the scoring rubric. Once codes were 
established, a categorization manual was constructed. Two graduate students, who were not otherwise 
familiar with the research project, were trained on the coding scheme, and inter-rater reliability was obtained 
on 20% of the data with greater than 90 percent agreement. Collectively, the maximum possible score on the 
conceptual knowledge survey was nine points. A paired-samples t-test was conducted to determine statistical 
significance in conceptual knowledge gain within each condition after treatment. A between-subjects 
ANCOVA was conducted to determine whether the main condition effect was significant. For this survey, we 
weren't able to conduct a multilevel analysis due to the small sample size but we are aware of the non- 
independence issues that can potentially impact results. However, as Cress (2008) notes, computer supported 
collaborative learning environments are designed to be influenced by group interactions and we intentionally, 
designed the conditions to understand the impact of learning scaffolds of which group interaction is one. 
Worksheets : A worksheet was used to gather data from all participating students, regardless of condition. 
Students in Conditions 1, 2, and 3 completed the worksheet individually immediately after they finished 
interacting with the device. They moved away from the device to be seated at tables for the task. Students in 
Condition 6 actually completed the worksheet during the group experience with a single worksheet on a 
clipboard being used for their collective response. Each group of three was allowed to negotiate the work 
process for themselves. For example, some groups elected a single scribe while others passed the sheet 
around to share the scribe duties. The worksheet required them to respond to this open-ended question: 
“What are you supposed to learn by using this device?” This question was intended to elicit responses that 
demonstrated ability to theorize from the interaction with the device, i.e., understanding of electrical circuits 
and how the human body functioned as a conductor. Responses were coded on four levels from no 
understanding (0) to complete understanding (3). Categorization manual construction, coding and reliability 
activities similar to those performed for the open-ended conceptual knowledge question were conducted. An 
ANOVA was conducted on the worksheet data set to determine whether there was a statistically significant 
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difference in responses between the conditions and a post-hoc Tukey HSD comparison was conducted to 
determine the source of the difference. Interviews : In order to investigate how knowledge-building scaffolds 
impacted the nature of the intervention, 66 students (-20% of the population) from C5 and C6 were randomly 
selected for short interviews to determine the utility of the scaffolds following their interaction with the 
device. We asked the students to evaluate each of the scaffolds they encountered in order to understand 
which were more or less impactful on their experience. 


2.3 Results 

2.3.1 Conceptual Knowledge 

Across the conditions, pre- and post-survey means ranged between 2.903 (pre-survey, C3) to 4.014 (post- 
survey, C4) out of a possible score of 9. Table 2 shows the results of a paired-samples t-test conducted within 
each condition. The table shows that gains in all conditions except Cl were statistically significant. Students 
in C4 demonstrated the greatest gains, with students in C3 a close second. However a between-subjects 
ANCOVA using the pre-test scores as covariate showed that there was no significant difference in the 
outcomes between six groups, F( 5, 300) = .546, p=. 741. 

Table 2. Results of Paired-Samples T-Test Comparing Means within Conditions 


Condition 

Mean Difference 

SD 

t 

df 

Sig. 

(2-tailed) 

1 

.337 

1.691 

1.187 

29 

.245 

2 

.484 

1.313 

2.051 

30 

.049* 

3 

.677 

1.833 

2.058 

30 

.048* 

4 

.681 

1.362 

4.241 

71 

.001* 

5 

.366 

1.355 

2.278 

70 

.026* 

6 

.472 

1.538 

2.605 

71 

.011* 


* p<0.05 


2.3.2 Cognitive Understanding 

Students were asked What are you supposed to learn by using this device? Response scores ranged from 
2.233 (low emergent understanding) to 2.800 (nearly partial understanding). Figure 2 shows an increasing 
trend from Cl to C6 in the means of theorizing abilities. 


Worksheet Responses: Ability to 



Cl C2 C3 C4 C5 C6 

Condition 


Figure 2. Mean scores for ability to theorize for conditions 1-6 

An ANOVA comparing mean scores showed a significant difference in results, FY5 (=2.234, p=. 052. A 
post-hoc Tukey analysis attributed the difference to the higher mean of C6 which was marginally 
significantly higher than Cl (p=.056). Differences between all other conditions were statistically not 
significant. 
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2.3.3 Impact of Knowledge-Building Scaffolds 

C5 and C6 students were asked which knowledge-building scaffolds were helpful. Figure 3 graphs their 
responses. 


Frequency of student Responses 
to Scaffold Helpfulness (n=66) 


65 



Figure 3. Frequency of student responses indicating helpfulness of scaffolds 

Many students found both group work and the worksheet helpful while fewer than 1/2 found the 
directions on how to work together helpful. Greater than 2/3 indicated that the knowledge-building prompts 
were helpful and greater than 1/2 indicated that the bank of previous answers was helpful. 


3. CONCLUSION 

Based on the conceptual gains shown in Table 2, this study supports previous findings (Yoon et ah, in press- 
a; in press-b) that digital augmentations can serve as valuable scaffolds for conceptual learning. Furthermore, 
where C3 and C4 students had the highest mean difference, we found that worksheet questions, which we 
consider to be a form of graphic panel (Serrel, 2006) when posted , can also support conceptual gains. We 
also found this to be true in our previous studies. When interviewed, C5 and C6 students said that working in 
groups was the most helpful scaffold besides worksheets, which suggests that the combination of group work 
and posted worksheets can assist in learning science concepts. 

Although the knowledge prompts and the bank of previous answers were selected fewer times as helpful 
scaffolds than working in groups and worksheets, the fact that a good portion of the population endorsed 
these more formal scaffolds is encouraging. Also, the result that C6 students had the highest score in higher- 
order reasoning somewhat supports an argument for their continued use. Flowever, the main reason for 
undertaking this research was to understand the extent to which knowledge-building scaffolds enhance the 
museum learning experience and which are most impactful. We hypothesized that “over-formalization” in an 
informal environment may negatively impact learning. This would include the use of scaffolds other than the 
kinds of graphic panels that are normally encountered. Our results indicate that greater conceptual gains 
occurred when scaffolds were used in less formal ways, i.e., when the worksheet questions were posted (C3 
and C4) and when they looked more like graphic panels typically found in museum settings, which in part 
satisfies our hypothesis. The fact that C6 students demonstrated greater theorizing ability suggests, however, 
that there may be a trade-off in what students can leant in informal environments in which learning takes 
place in short episodes. That is, we may be able to improve student's conceptual understanding somewhat at 
the expense of improved cognitive gains if we consider the unique characteristics of informal settings. 

Repeat visitation and use of an exhibit space could counteract this pattern as conceptual understanding 
can grow over time and later become more formalized for the sake of cognitive gain. Through repeated use of 
an exhibit device, especially with a variety of knowledge-building scaffolds available, students find multiple 
learning pathways. The habit of repeated use of an exhibit remains a challenge given the episodic nature of 
visitation. On possible way to address this issue is a distributed virtual social learning space that can extend 
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engagement with topics from multiple exhibit devices outside of the museum’s physical space. Such a 
collaborative space is currently in development for testing in future phases of our research. 
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ABSTRACT 

Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the business 
strategy and performance measurement. Many organizations around the world are using the BSC to define, implement 
and manage strategy. Nevertheless there exist studies that identify problems and limitations associated with the 
implementation and use of the BSC. Those studies show in general terms that managers do not understand the BSC as the 
measures and perspectives in use are fairly independent, and do not always mirror the recommended cause-and-effect 
logic included in the systems perspective of the BSC approach. This article addresses the effectiveness of teaching the 
Balanced Scorecard by means of business simulation. An experiment that uses a business simulator is performed for 
testing a set of hypotheses about the influence of simulation on the students’ understanding of the BSC. The simulation 
experience was specifically designed by the authors to promote understanding of the BSC concepts. Student feedback and 
assessment showed that the simulation significantly enhanced the understanding of the BSC concepts related to the 
strategic management and double-loop learning processes and the systems perspective. Results also suggested that 
understanding of those BSC concepts positively influences simulation performance. 


KEYWORDS 

Balanced Scorecard, Business Simulation, Training, Business Education, Teaching Methods. 


1. INTRODUCTION 

Kaplan and Norton (1992) introduced the Balanced Scorecard (BSC) with the aim of overcoming strategic 
management limitations of the traditional performance measurement systems. The BSC approach features a 
mix of lead (performance drivers) and lag (outcome measures) indicators, financial and non-financial 
measures. These are categorized as follows: (i) financial, (ii) customer, (iii) process and (iv) learning and 
growth. The BSC tool helps managers monitor actual financial and market performance, evaluate the results 
of short-term processing actions, monitor the intangible development of competencies that will drive future 
financial performance and assess the progress of implementing corporate strategy. To support managers in 
developing a cause-and-effect perspective and to better understand the business system in which they 
participate, Kaplan and Norton (2001) developed the strategy map concept as a complementary tool to the 
BSC approach. The strategy map links the performance indicators in a causal chain (causal diagram) that 
helps managers to translate, test and communicate their understanding of the business system. 

Managers make decisions and learn in the context of feedback loops (Forrester, 1961). In single-loop 
learning, managers compare information about the state of a real system to pre-established goals, perceive 
deviations between desired and actual states, and make the decisions they believe will move the system 
towards the desired state. Single-loop learning does not change the managers’ mental models. A mental 
model is a conceptual representation of the structure of an external system used by people to describe, 
explain and predict a system’s behavior (Craik, 1943, Johnson-Laird, 1983). In double-loop learning, 
information about the business system is not only used to make decisions within the context of existing 
frames, but also feeds back to modify the managers’ mental models (Argyris, 1976). As their mental models 
change, managers define new strategies and policies. 

It is clear that the BSC approach is consistent with the systems and feedback learning perspective of 
business management and performance measurement. Kaplan and Norton (2001) argue that the BSC 
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approach supports double -loop learning that facilitates managerial strategic learning, leading to better 
performance. In a continual process, managers use the balanced scorecard and the strategy map to re-evaluate 
the assumptions used in the previous strategy. They review the assumed cause-and-effect relationships and 
identify new ones. Then they improve their understanding of the business system and they determine a new 
strategy (Kaplan and Norton, 2001). In other words, BSC triggers a process by which managers can make 
explicit improvements to their mental models of the business system. They adapt the company strategy and 
define the new short and middle term objectives by simulating their mental models to infer the future 
behavior of the business system. 

Many organizations around the world are using the BSC approach to define, implement and manage 
strategy. However, according to empirical research into the performance implications of the BSC, the 
positive contributions of the BSC have not been unambiguously confirmed. There exist studies in the field of 
management accounting research that identify problems and limitations associated with the BSC approach. 
The inadequate definition and utilization of the performance indicators has been highlighted as a main 
drawback of the BSC system (Lingle and Schiemann, 1996; Stivers et al., 1998; Ittner and Larcker, 1998, 
2003; Lipe and Salterio, 2000; Malmi, 2001; Speckbacher et al, 2003). Those studies show in general terms 
that the measures and perspectives in use are fairly independent, and do not always mirror the recommended 
cause-and-effect logic of the BSC approach. 

1.1 The use of Simulation for teaching the BSC 

Business simulators, business games, management simulators, microworlds, learning laboratory, interactive 
learning environments are some examples of terms used to describe computer-based simulations that have 
been proposed by researchers from many fields, e.g., management and decision science, psychology, 
education and computer science, as important tools to support management learning processes (Maier and 
Grobler, 2000). In a broad definition, business simulators for learning purposes comprise virtually everything 
connected with the learning process. An interactive learning environment contains more than a computer 
simulation model. A simulation model is embedded into a learning environment, which may also include 
case descriptions, presentations by a facilitator, modelling tools, background information, source material and 
working instructions (Davidsen and Spector 1997). 

Simulators are promising tools for teaching in business management domain as they expected to help 
students acquire knowledge. Students’ learning processes center around the exploration of the simulation 
model. They gather knowledge through an inquiry learning process as they make a broad analysis of the 
domain, generate hypothesis, experiment, interpret the outcomes, conclude about the validity of the 
hypothesis or form new ideas, and finally reflect on the domain (de long and van Joolingen, 2008). 
Additionally, by performing simulated tasks, learners understand through experiential processes how 
concepts are applied and why they are useful, thus enabling new learning to be more easily (Cannon-Bowers 
and Bowers, 2008). Other specific advantages have been pointed out. An often named advantage of computer 
simulations is the “compression of time” as they instantly show the results of decisions a user has made. 
Simulators help subjects leverage their domain-rich knowledge by allowing them to play through simulated 
years, reflect on their actions, modify their mental models, then repeat the process. By compressing time, 
business simulators can accelerate learning by enabling them to conduct many such cycles of action and 
reflection (Bakken et al, 1994). Providing a safe environment is at the same time an important advantage of 
simulators (Sterman 1994), which allows for experiential learning without that stress-related obstacles that 
are met in reality. However, subjects may tend to take more risks than in real-life decisions. Also, computer 
simulations of business systems address objectively certain special management issues and try to abstract 
from details and isolating from confounding factors (Isaacs and Senge, 1994). This abstraction allows a focus 
on the learning of important and specific business themes. 

As mentioned in previous section, several studies show in general terms that managers do not understand 
the fundamental concepts of the BSC approach, in particular those connected with the systems and feedback 
learning perspective. Thus, appropriate teaching and training methodologies must be designed in order to 
improve subjects learning and comprehension of those concepts. The present study addresses the 
effectiveness of teaching the Balanced Scorecard by means of business simulation. An experiment that uses a 
business simulator is performed for testing a set of hypotheses about the influence of simulation on the 
subjects’ understanding of the BSC. The effectiveness and relevance of simulators for learning is intuitively 
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acknowledged by many researchers. However, there exists scarce research about the effectiveness of 
computer-based simulations to support learning. Thus, this study also aims to be a contribution to the 
research issue of “effectiveness of computer-based simulations to support learning” not been solved so far. 


2. RESEARCH HYPOTHESES 

This research focuses on how business simulation facilitates the teaching of the Balanced Scorecard 
approach, leading to enhanced students’ understanding of its main concepts. The analysis of the subjects’ 
understanding of the BSC concepts considers two components: the performance measurement system and the 
systems and strategic learning perspective. The performance measurement system associated with the BSC 
approach is viewed as a comprehensive structure of performance indicators that combine both financial and 
non-fmancial measures. By using a BSC performance measurement system, managers access more relevant 
information to interact with the business environment, and they acquire a sharper understanding of the impact 
of their decisions. The systems and strategic learning perspective - The BSC approach is consistent with the 
systems and strategic learning perspective. By using the BSC strategy map tool, managers will benefit from 
more effective double-loop learning as they review the critical cause-and-effect relations through a process 
that externalizes and improves their mental models of the business system. 

It is assumed that by teaching the BSC through business simulation, students will benefit from more 
effective understanding of the BSC performance measurement system concepts. Hypothesis 1: The use of 
simulation improves the level of understanding of the BSC performance measurement system. 

It is assumed that by teaching the BSC through business simulation, students will benefit from more 
effective understanding of the systems and strategic learning perspective of the BSC approach. Hypothesis 2: 
The use of simulation improves the level of understanding of the BSC as a strategic learning system. 
Hypothesis 3: The use of simulation improves the level of perceived relevance of the strategic learning 
system concepts. 

This study also assumes that students perform better the business simulation task if they reach both higher 
level of understanding of the BSC performance measurement system and higher level of understanding of the 
systems and strategic learning perspective of the BSC approach. Hypothesis 4: The level of understanding of 
the BSC performance measurement system is positively correlated with Performance. Hypothesis 5: The 
level of understanding of the BSC as a strategic learning system is positively correlated with Performance. 
Hypothesis 6: The level of perceived relevance of the strategic learning system concepts is positively 
correlated with Performance. 


3. METHOD 

The hypotheses defined in this research and presented in the previous section were tested with a simulation- 
based experiment. This section presents an overview of the simulator, describes the subjects and the 
experiment conditions, and overviews the research variables. 

The business simulator was built by incorporating the same system dynamics model that had been used in 
previous research (Capelo and Dias, 2009). The participants run a realistic simulator of a wireless 
telecommunications firm by making critical decisions every six months for a simulation period of seven 
years. The participant objective was to develop critical and interrelated resources at appropriate rates and 
levels in order to gain and retain customers, operate efficiently, and maximize value creation. To succeed in 
this simulation task, participants had to identify and understand the cause-and-effect relationships among 
critical variables. The simulator provides an interface that represents a balanced scorecard that includes a set 
of leading and lagging, financial and non-financial indicators that are graphically separated into four sections 
related to the four perspectives associated with the BSC approach. The simulation task also involves drawing 
and reviewing a strategy map. The participants produced and reviewed a strategy map linking the critical 
concepts as were found in the simulation model. This strategy map represents the participants’ understanding 
of the structure of the simulated business system. 
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The subjects consisted of 74 undergraduate students. All the participants were familiar with basic BSC 
concepts as they attended previous classes on the BSC approach. The participants had no experience with the 
simulator and they also had no prior specific knowledge about wireless telecommunications businesses. 

All participants were given a full experimental guide including: description and objective of the 
simulation task; case text; instructions for accessing and starting the simulator on the computer network; 
instructions for running the simulator; sheets for strategy map review. The decisions made on the simulation 
and its results were automatically stored in a protected spreadsheet on the participant’s computer. 



Figure 1 . Experimental procedure 

The firm was run by using the balanced scorecard and the strategy map. They analyzed business status 
using the simulator interface, used this information to review the strategy and objectives and decision 
making, and then repeated the process. The experiment procedure involved four sessions and had the 
following steps (Figure 1). Session 1: a) The participants answered a questionnaire on BSC approach. This 
questionnaire, comprising twenty five multiple choice questions about the BSC, and a ten-point scale to 
evaluate the relative relevance of ten critical concepts (five regarding the performance measurement system, 
and other five regarding the systems and strategic learning perspective), captured their initial level of 
comprehension of the BSC approach. Session 2: b) The participants read the introduction with the overall 
description and the objectives of the simulation and the business case and were instructed to raise any 
questions they had as they proceeded with the case, c) The participants read the instructions for accessing, 
starting and running the simulator and they were given oral instructions with examples to show simulator 
operation, d) A first simulation was conducted to familiarize participants with the game interfaces and 
commands, and at this point they were instructed to ask for help at any time. Session 3: e) The participants 
received instruction in how to draw the strategy map, and how to review it by cutting or inserting links 
between the indicators and defining the arrows that indicated the cause-and-effect relationships, f) The 
participants drew the initial strategy map. g) The participants performed the definitive simulation. They were 
also asked to review the causal diagram (strategy map). They cut or inserted links so that the causal diagram 
accurately expressed their latest understanding of the simulated business system. The participants were also 
encouraged to use the strategy map to reflect on strategy, objectives, and decisions. Session 4: h) The 
participants filled out the same questionnaire that was used on the session 1 in order to capture their ultimate 
understanding of the certain critical concepts of BSC approach. Finally, the participants described their 
simulation experience in a written report which was part of the evaluation process. 

3.1 Research Variables 

Level of understanding of the BSC performance measurement system (LUPMS-BS and LUPMS-AS). These 
variables were measured in terms of the evaluation score of participants in answering a questionnaire, before 
the simulation experiment (LUPMS-BS) and after the simulation experiment (LUPMS-AS). 

Level of understanding of the BSC as a strategic learning system (LUSLS-BS and LUSLS-AS). These 
variables were measured in terms of the evaluation score of participants in answering a questionnaire before 
the simulation experiment (LUSLS-BS) and after the simulation experiment (LUSLS-AS). 

Level of relevance of the performance measurement system concepts (LRPMS-BS, LRPMS-AS and 
LRPMS-CA). These variables represent the students’ perception on the relative importance of the 
performance measurement system of the BSC approach. LRPMS-BS and LRPMS-AS were obtained from 
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the questionnaire answered by participants, before and after the simulation experiment. The variable 
LRPMS-CA results from a quantitative content analysis of the final report. 

Level of relevance of the strategic learning system concepts (LRSLS-BS, LRSLS-AS and LRSLS-CA). 
These variables represent the students’ perception on the relative importance of the systems and strategic 
learning perspective of the BSC approach. They also result from the questionnaire answered by participants, 
before and after the simulation experiment. The variable LRSLS-CA was also obtained from quantitative 
content analysis of the final report. 

Performance - Task performance was measured by total financial value creation. 


4. RESULTS AND DISCUSSION 

Table 1 shows the results of statistical testing to identify differences in means between stages. Unexpectedly, 
after simulation the participants showed on average a lower level of understanding of the BSC performance 
measurement system (LUPMS). However, the mean values of LUPMS for after simulation stage were not 
significantly different from the equivalent values for before simulation stage (mean difference=-0.32, p=315). 
On average, after simulation, the participants showed a better level of understanding of the BSC as a strategic 
learning system (LUSLS), a lower level of relevance of the performance measurement system concepts 
(LRPMS), and a better level of relevance of the strategic learning system concepts (LRSLS). Table 1 shows 
that those differences are significant at p<0.01 (mean difference LUSLS=0.188, p<0.001; mean difference 
LRPMS=-3.39, pcO.OOl; LRSLS=3.63, p=0.001). 


Table 1. Tests of significance for differences in means between the stages 


Variable 


After Simulation - Before Simulation 


Mean 

Difference 

Standard Deviation 

Significance 

P 

LUPMS 

-0.32 

2.44 

0.315 

LUSLS 

1.88* 

2.85 

0.000 

LRPMS 

-3.39* 

6.96 

0.000 

LRSLS 

3.63* 

7.65 

0.001 


*p<0.01 


Table 2. Tests of significance for differences in means for BSC concepts between stages 



After Simulation - Before Simulation 

BSC Concept 

Mean 

Difference 

Standard Deviation 

Significance 

P 

Concepts included in LRPMS 
C 1 - BSC perspectives 

-2.20** 

3.49 

0.000 

C 2 - Financial measures 

-0.85* 

2.96 

0.032 

C 3 - Non-financial measures 

-0.03 

3.30 

0.937 

C 4 - Objectives and targets 

-0.25 

3.42 

0.571 

C 5 - Key initiatives 

-0.05 

2.96 

0.895 

Concepts included in LRSLS 
C 6 - Systems and dynamical view 

0.49 

4.20 

0.372 

C 7 - BSC strategy map 

1.00* 

3.77 

0.046 

C 8 - Understanding and validation of the cause- 

1.47** 

2.87 

0.000 

and-effect relationships 

C 9 - Inference about future performance 

0.22 

3.62 

0.642 

CIO - Double- loop learning process 

0.44 

3.80 

0.376 


*p<0.05; **p<0.01 


Table 2 shows the results of statistical testing to identify differences in means between the stage before 
simulation and the stage after simulation for each BSC concept. The lower mean value of LRPMS revealed 
by participants after simulation was mainly due to Cl - BSC perspectives (mean difference =-2.20, p<0.001) 
and C2 - Financial measures (mean difference =-0.85, p<0.05) concepts. These results suggest that after 
simulation the participants changed their perception about the importance of those concepts as they rated on 
average a lower level. Results in table 2 also suggest that the better mean value of LRSLS after simulation 
was mainly due to C7 - BSC strategy map (mean difference =1.00, p<0.05) and C8 - Understanding and 
validation of the cause-and-effect relationships (mean difference =1.47, p<0.001 ) concepts. Thus, on 
opposite, it seems that the simulation experience made the participants changing their view on the relevance 
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of these concepts as they rated on average a higher level. The relevance of those BSC concepts was also 
evaluated by means of a quantitative content analysis of the final report (table 3). According to that analysis, 
the participants perceive on average that C7 (BSC strategy map), C8 (understanding and validation of the 
cause-and-effect relationships) and CIO (Double-loop learning process) are the most relevant BSC concepts. 

In order to evaluate the relationship between Performance and each variable, a bivariate correlation 
analysis was conducted. As it can be seen in table 4, there is some correlation between Performance and 
LUPMS-AS (Con-0.245, p=0.053). However, that correlation is not significant (at p<0.05). Performance is 
significant correlated (at p<0.05) with LUSLS-AS, and with LRSLS-CA. 


Table 3. Results of quantitative content analysis of the simulation reports 


BSC Concept 

Quantitative Content Analysis of 
Simulation Reports 
Number of 

Mentions % 

Concepts included in LRPMS 

63 

36.0% 

C 1 - BSC perspectives 

16 

9.1% 

C 2 - Financial measures 

7 

4.0% 

C 3 - Non-financial measures 

18 

10.3% 

C 4 - Objectives and targets 

7 

4.0% 

C 5 - Key initiatives 

15 

8.6% 

Concepts included in LRSLS 

112 

64.0% 

C 6 - Systems and dynamical view 

6 

3.4% 

C 7 - BSC strategy map 

31 

17.7% 

C 8 - Understanding and validation of the cause- 
and-effect relationships 

32 

18.3% 

C 9 - Inference about future performance 

19 

10.9% 

CIO - Double-loop learning process 

24 

13.7% 


Table 4. Analysis of correlation (Pearson) 


Variables 

Correlation Analysis 

Performance 


Pearson Correlation 

Significance 

LUPMS-AS 

0.245 

0.053 

LUSLS-AS 

0.298* 

0.018 

LRPMS-AS 

0.019 

0.882 

LRSLS-AS 

-0.030 

0.817 

LRPMS-CA 

0.186 

0.144 

LRSLS-CA 

0.299* 

0.017 


*p<0.05 


Table 5 shows the results of multivariate regression analyses of Performance on the independent variables 
(model 1). The regression model was then refined (model 5) by performing a stepwise regression in order to 
exclude the variables that did not seem to significantly explain the dependent variable and to preserve the 
most significant explanatory variables. Regression analysis of Performance on the most significant 
independent variables shows significant effects for LUPMS-AS and LRSLS-CA. 

These results confirm five of the six hypotheses (table 6). The present study does not support Hypothesis 
HI as from table 1 the mean values of LUPMS for before and after simulation stages were not significantly 
different. On average, the participants in after simulation stage showed better LUSLS and LRSLS, with 
significant differences evident in Table 1, supporting Hypothesis H2 and Hypothesis H3. The correlation 
analysis (table 4) showed no significant relationship between LUPMS-AS and Performance. However, the 
results of multivariate regression analysis of Performance on the independent variables (model 5 - table 5) 
indicates a significant effect for LUPMS-AS. Consequently, the results provide support of Hypothesis H4. 
The results from correlation analysis (table 4) provide support of Hypothesis H5 as it was found a significant 
correlation between LUSLS-AS and Performance. LRSLS was measured through two processes: data 
obtained from the questionnaires answered by participants after simulation (variable LRSLS-AS) and data 
extracted from quantitative content analysis of participants’ simulation report (variable LRSLS-CA). 
Correlation between LRSLS-AS and Performance is not significant. However, results reveal a significant 
correlation between LRSLS-CA and Performance. Thus, the results provide support of Hypothesis H6. 
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Table 5. Model 1: regression results for all independent variables; Model 5: regression results obtained through a 

stepwise procedure 




Dependent Variable 

Independent 

Variables 


Performance 

Standardized Beta Significance 

Model 1 
LUPMS - AS 

0.226 

0.088 

LUSLS - AS 

0.144 

0.273 

LRPMS - AS 

0.111 

0.783 

LRSLS - AS 

0.053 

0.894 

LRPMS - CA 

0.176 

0.151 

LRSLS - CA 

0.336** 

0.009 

Model 5 
LUPMS - AS 

0.292* 

0.016 

LRSLS - CA 

0.340** 

0.006 


*p<0.05; **p<0.01 


Table 6. Summary of Hypothesis Testing 


Hypotheses 

Description 

Results 

HI 

The use of simulation improves the level of understanding of the BSC 
performance measurement system 

Not supported 

H2 

The use of simulation improves the level of understanding of the BSC as 
a strategic learning system 

Supported 

H3 

The use of simulation improves the level of perceived relevance of the 
strategic learning system concepts 

Supported 

H4 

The level of understanding of the BSC performance measurement system 
is positively correlated with Performance 

Supported 

H5 

The level of understanding of the BSC as a strategic learning system is 
positively correlated with Performance 

Supported 

H6 

The level of perceived relevance of the strategic learning system concepts 
is positively correlated with Performance 

Supported 


4. CONCLUSION 

Kaplan and Norton introduced the Balanced Scorecard (BSC) which is based on a systems perspective of the 
business strategy and performance measurement. Many organizations around the world are using the BSC to 
define, implement and manage strategy. Nevertheless there exist studies that show in general terms that 
managers do not understand the BSC as the measures and perspectives in use are fairly independent, and do 
not always mirror the recommended cause-and-effect logic included in the systems perspective of the BSC 
approach. Thus, appropriate teaching and training methodologies must be designed in order to improve 
subjects learning and comprehension of those concepts. The present study addresses the effectiveness of 
teaching the Balanced Scorecard by means of business simulation. An experiment that uses a business 
simulator is performed for testing a set of hypotheses about the influence of simulation on the subjects’ 
understanding of the BSC. The simulation methodology significantly enhanced the understanding of the BSC 
concepts related to the strategic management and double-loop learning processes and the systems 
perspective. Results also suggested that understanding of those BSC concepts positively influences 
simulation performance. 

The findings confirm our assumptions on the effectiveness of using simulation for teaching the BSC 
approach as a strategic management system (involving balanced scorecards and strategy maps as suggested 
by Kaplan and Norton). The simulator operates by participants developing and combining critical resources 
in appropriate levels in order to attract and retain customers, while running an efficient company. To reach 
this goal, the participants must understand the interdependence among critical resources and variables of the 
business case and they must combine these effectively. As students create a causal model (the BSC strategy 
map) representing the critical cause-and-effect relations of the business case, and use that causal model and 
the feedback information from the BSC performance system for running the firm, they develop an effective 
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understanding about the meaningless and usefulness of the BSC concepts related to the strategic management 
and double-loop learning processes. 
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ABSTRACT 

The following paper examines the results of a research study in which a virtual world. Quest Atlantis (QA), was used to 
engage students in exploratory learning to teach about water quality issues. The main aim of the research was to find out 
how new digital learning environments and educational technology, such as virtual worlds, can be introduced in schools; 
what will be the response of the administration, teachers, parents and students; what could be the best teaching 
methodology for these digital learning environments; and what could be possible hurdles and hindrances. The research 
was comprised of comparison between different teaching methods based on student outcomes and the results of a survey 
which was conducted at the end of the study. The research was too big to include all results in this paper thus the findings 
of the survey and observations of the researcher are presented here. The survey was answered by students (N=33) and it 
included questions about the learning environment, perceptions about the learning process and problems that the students 
might have encountered during the study. During the study, the students were divided into three different groups, or 
classes, and each group studied using a different teaching method. Thus, there were three teaching methodologies or 
learning strategies: direct instruction and using QA; group discussions in class and using QA; and self-exploration using 
QA. The results are presented on an overall basis and also on the basis of these teaching methods as it could have 
influenced the answers given by the students. The results of the survey shows that majority of the students liked learning 
through the virtual world, Quest Atlantis in this case, and would like to continue using it both at home and at their school. 
Irrespective of the teaching method used, they perceived that they learned a lot about water quality and think that they 
could not have learned more in a traditional class. The biggest hurdles or problems reported by students were: working in 
pairs; wrong time to conduct this research; short session times; and technical problems related to computers and the 
internet. 


KEYWORDS 

Virtual worlds, digital learning environments, exploratory learning, teaching methodologies. 


1. INTRODUCTION 

Educators all over the world throughout the history have been striving to make the process of learning more 
engaging for the learners. For this purpose, educators also try to make use of any new technology especially 
those which are popular among learners. However, integration of those technologies into the settings of 
formal learning institutions, schools and universities, can be a challenge. Virtual worlds are one of those 
technologies for which many researchers have proved its potential for education. For example, Iqbal et al 
(2010: 3199) presented learning gains in virtual worlds as is presented by many researchers. Barab et al 
(2008), Barab et al (2009), Hickey et al (2009), Arici (2008) have shown learning gains when learners used 
the virtual world of Quest Atlantis (QA). Similarly, Ketelhut et al (2005) has also shown learning gains using 
River City. 

However, many researchers have indicated that there are many obstacles which hinder the integration of 
virtual worlds in the school or university settings. Dickey (2011: 15-17) discusses that visual representation, 
security and costs were main hindrances as perceived by the teachers in his research. Klopfer et al (2009: 18) 
presents 9 barriers to adoption for learning games which are curriculum requirements, attitudes, logistics, 
support for teachers, assessment, evidence, uses of games, limited view and social and cultural structures. 
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Technical problems, steep learning curve, student security and university liability was identified to be the 
major concerns in Second Life (Woods, 2010: 105-108). Koh et al (2012: 56) found that insufficient time, 
limited resources, high costs, non-relevance to curriculum, reactions from parents and no or less support from 
school are key reasons for not using games in schools in Singapore as perceived by teachers. Ryan (2008), a 
PhD student at Lancaster University, presented sixteen ways to incorporate Second life into a classroom. 

Most of this research is conducted in western countries. These problems could amplify or more problems 
can arise in developing countries. Here a study is presented which is significant as it compares different 
teaching methods and also because it was carried out in Pakistan to find out the relevant problems in 
integrating virtual worlds in a developing world context. This paper presents a research in which students 
learned in QA through three different teaching methods. The research was focused on comparison of learning 
gains and experiences between these three different teaching methods. Here we present the experiences of the 
students and hindrances that they faced which were collected as survey answers. 

1.1 About the Virtual World 

The virtual world used in this research is Quest Atlantis (QA) which is currently maintained under the 
Atlantis Remixed Project (Atlantis Remixed Project: Online). QA engages students in, what Atlantis 
Remixed Project team defines as transformational play through content that is comprised of online, in QA 
virtual world and on websites, and off-line activities. The Atlantis Remixed Project team claims to have 
demonstrated learning gains in science, language arts and social studies. 

1.1.1 The Topic 

The topic chosen for this research was water quality which is known as ‘Taiga: the water quality unit’ in QA. 
The researchers wanted to choose a topic which was unfamiliar with the students in Pakistan. None of the 
students who took part in this study had ever learnt about water quality although some of them were familiar 
with the concepts of pH, acids and bases. They were unaware of the effects that water quality can have on the 
ecosystem. 

The learning takes place in QA at ‘Taiga National Park' where the park ranger had asked the students to 
help him to identify the reasons for declining number of all species of fish in ‘Taiga River’ (Taiga Unit Plan: 
Online). 



Figure 1. Map of ‘Taiga National Park’. 
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Along the banks of Taiga River live three main groups of people. ‘Mulu’ indigenous people use the river 
to grow crops and raise different animals. ‘Build-Rite Lumber Company’ cuts and logs the trees in the park. 
Lastly, there is a fishing company, ‘K-Fly Fishing Company’, which arranges fishing tours. ‘Taiga National 
Park' also has a ranger station, lab and 3 water monitoring stations as is shown in Figure 1 . 

The material used in this research was provided by the Atlantis Remixed Project team in ‘Taiga Unit 
Plan' (Taiga Unit Plan: Online) which is available online. 

1.1.2 The Missions 

There are five missions in ‘Taiga: the water quality unit’ but due to time constraints it was decided that the 
students will try to solve first three missions including an introduction week to using QA. The educational 
year in most of the schools in Karachi, Pakistan, runs from April to March. This study started in January and 
as the final exams were approaching thus the school administration was not ready to give more time. 

The first mission is called ‘Getting a handle on Taiga’ and in this mission the students meet with all the 
stake holders of the Taiga National Park. They talk with all of them and get their respective versions of the 
possible problems of water quality and who may be responsible for the situation. In doing this, the students 
start understanding that it’s not as straight as it might look initially and that the situation is complex. At the 
end of the mission the students have to fix responsibility for declining population of fish. 

The students continue to talk with different people in the second mission, ‘Digging deeper', and take 
photos of Taiga River. Then they compare the photos taken and try to analyze them in order to come up with 
a hypothesis. Students learn about distinguishing between guess and hypothesis. 

In mission 3 students collect water samples from three different sampling sites and then take it to the park 
lab to analyze it. This mission is called ‘Building a case’ and students learn about and analyze 6 water quality 
indicators which are pH, nitrates, phosphates, dissolved oxygen, turbidity and temperature. Based on the 
results of the analysis the students inform the park ranger about the group responsible for the problems. 


2. METHODOLOGY 

The research had mainly two parts. Firstly, to compare different teaching methods based on student out 
comes. In order to do this the students (n=48; 27 male, 21 female) were divided into three classes which were 
organized according to three different teaching methods. Teaching method 1 (TMl) had 16 students. 
Teaching method 2 (TM2) had 15 students. While, teaching method 3 (TM3) was comprised of 17 students. 
The teaching methods are briefed later in this section. 

The second part was to get an insight into student's experiences by asking them to answer a survey. The 
total number of students who answered the survey was 33 (19 male, 14 female). Out of that, 13 were from 
TMl, 12 were from TM2 and 8 were from TM3. 

Although, the survey was supposed to be conducted online but the teachers suggested that it will be easier 
and quicker to do it on paper. Thus, the students took the survey on paper and later on their answers were put 
in the computer. 

Throughout the study the researcher observed the whole learning process and had many discussions with 
owners, administration and teachers. 

2.1 Context of Research 

The research was carried out in a privately owned school in Pakistan by a university in Europe (Nordic 
countries. The name of the institution is not mentioned to maintain the anonymity of the author). On the 
contrary to Europe, the schools in the educational sector in Pakistan can be divided into government-owned 
or privately-owned. Privately-owned schools in Pakistan are considered to be of better quality as compared to 
the government-owned schools. Therefore, mostly all privately-owned schools, except those run by non- 
profit or charity organizations, charge a monthly fee to meet their expenses and to generate profit on their 
investments. In return they provide better facilities, teachers and resources. 

This context of research is important to mention here because the researcher had to convince owners of 
the school, administration, teachers and parents about the importance of the research in order to justify using 
the resources of the school. In this research, certificates of participation were given to the school, teachers 
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and students. The school was also given a complimentary gift by the research unit which conducted this 
study. The students were so enthusiastic when they heard about this study that they played a crucial role in 
convincing their parents. 

Another aspect that has to be mentioned here is that the teachers and a group of volunteers went through a 
week of training period in which they used Quest Atlantis as the teachers were not accustomed to playing 
video games and/or using virtual worlds. This proved to be very helpful when the research was carried out. 

2.2 Teaching Methods 

All students who took part in the study were divided into three groups or classes and each group or class 
learned through a different teaching method or learning method. These three methods can also be called 
classroom implementation strategies. These teaching methods are mentioned as under. 

2.2.1 Teaching Method 1 (TM1): Explanatory Instruction and Exploratory Inquiry-Based 
Learning in QA 

In this teaching method the students (n=16) learned through traditional direct instruction method where the 
teacher explained the topic and the main concepts in the class. After that, students went to the computer lab 
to use QA in pairs in which they undertook the missions. 

2.2.2 Teaching Method 2 (TM2): Group Discussion and Exploratory Inquiry-Based Learning 
in QA 

The students (n=15) used QA in the computer lab in groups of two (pairs) however one of the groups had 
three students. Afterwards, the students went into class where they took part in group discussions where they 
discussed the topic in larger groups consisting of three to four students and then shared their understanding 
with the whole class. 

2.2.3 Teaching Method 3 (TM3): Self-Exploration Through Guided Inquiry Instruction Using 
QA 

There were 17 students who were divided into groups of two (pairs) however one of the groups had three 
students in this teaching method. The students learned through self-exploration by using QA on their own 
and they were allowed to ask questions and get guidance from teachers if they wish to. 

2.3 Survey Design 

The survey can be hypothetically divided into three main parts. As the survey was supposed to be taken at the 
end of the comparison study, thus the survey contained questions about the experience of learning through a 
virtual world, QA in this case. First question asked the students whether they liked learning through QA or 
not? They had to respond by choosing one of the following: ‘yes, it was very fantastic’, ‘yes, it was good', ‘it 
was ok’, ‘no, it was not good' and ‘not at all, it was really bad'. Another question was ‘would they like the 
school to continue using QA?’ And they had to choose between ‘no, not at all. It is useless’, ‘yes, it is very 
good educational software' and ‘may be. I am not sure'. Thus, the students had equal positive and negative 
and a neutral statement to choose from. The students were also asked whether they would be using QA at 
home or not. They answered by choosing ‘yes, definitely’, ‘yes, sometimes’, ‘no, never' and ‘I do not know’. 

The other part of the survey asked students about their perception of learning through QA. They were 
asked how much they perceive that they have learned in this research. The answer choices were: ‘I learned a 
lot’; ‘I learned something'; ‘I did not learned much’; ‘I learned nothing at all'; and ‘I do not know’. Thus, the 
students had equal positive and negative and a neutral statement to choose from. They were also asked 
whether they would prefer learning through ‘traditional class’, ‘through QA' or ‘through a combination of 
traditional class and QA'. There were two questions related to this theme, one about the perceived learning 
benefit and the other about enjoyment. 

In a question they were also asked about the features of QA that they liked. The features about which the 
students were asked about were: graphics; teacher guidance system; missions / quests; gaming elements; 
interface; customization of the avatar; story; topic; answering system and the learning process. 
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The last main part of the survey was about the difficulties, hurdles and hindrances that the students faced 
during the study. They were asked about the problems that might have occurred due to the way it was 
conducted. They were also inquired about the problems that they encountered in QA or technical problems 
about computers and internet. 


3. RESULTS AND DISCUSSION 

The total number of students who answered the survey was 33 out of the 48 students who took part in the 
research. The students mainly liked learning through QA in all teaching methods and they would like to 
continue learning from QA. Three questions specific to the missions in QA were also asked from the students 
in the survey but there results are not presented here as they do not fit in the theme of this article. Main 
results are discussed under the following sections. 

3.1 About Learning Process in QA 

The students were asked three questions about the learning process. First question inquired if they believe 
that they have learnt in this research or not. A total of 18 (54.55 %) students agreed that they had learnt a lot 
in this study which is more than half of the students as is shown in Table 1. Eight (24.24 %) students 
expressed that they learnt something. If we combine the number of students who expressed that they had 
learnt a lot with those said that they had learnt something, then the total number of students, 26 (78.79 %), 
who expressed that they learnt using QA is quite large. On the contrary, only 6 ( 18.18 %) students mentioned 
that they did not leant much. There can be many reasons for not being sure if they learnt something or 
nothing. One of that could be that the topic was totally unfamiliar to them and it was not covered in a 
tradition course book as is in Pakistani schools the curriculum is strictly taught through an assigned book. 
Thus, if they were not learning from a pre-assigned book they were not sure what and how much were they 
learning. Curriculum requirements are one of the barriers to integrate video games, and educational 
technology, as was indicated by Klopfer et al (2009: 18). Secondly, they were not made aware of the post-test 
results which they could have compared with their peers. 

Students of TM2, 8 (66.67 %), were more sure that they learnt a lot than students of TM1, 6 (46.15 %), or 
the students of TM3, 4 (50 %). Thus, one can infer that group discussions in class do result in a belief that 
one has leant a subject. This could be because the students get a chance to share and adjust what they have 
learnt through discussing with peers. 


Table 1. How much do you think you learned about water quality from Quest Atlantis? 



I learned a lot 

I learned 
something 

I did not learned 
much 

I learned nothing 
at all 

I do not know 

TM1 (n=13) 

6(46.15 %) 

3 (23.08 %) 

3 (23.08 %) 

0 

1 (7.69 %) 

TM2 (n=12) 

8 ( 66.67 %) 

3 (25 %) 

1 (8.33 %) 

0 

0 

TM3 (n=8) 

4 (50 %) 

2 (25 %) 

2 (25 %) 

0 

0 

Total (n=33) 

18 ( 54.55 %) 

8 (24.24 %) 

6(18.18%) 

0 

1 (3.03 %) 


The students were also asked to specify in which of the following methods they could have learnt the 
most about water quality: traditional class learning; learning through QA; or a combination of both traditional 
class and QA. This was asked to find out their preferences after taking a course which involved a virtual 
world, in this case QA. There were 32 students who answered this question. Majority of the students, 22 
(68.75 %) as is shown in Table 2, expressed that they could have learnt the most through QA across all 
teaching methods. Students of TM3, 7 (87.5 %) out of 8, believed the most that they could not have learnt 
more than learning through QA. A majority of students of TM1, 8 (66.67 %), and TM2, 7 (58.33 %), also 
thought the same. 
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Table 2. In which of the following method do you think you could have learned the most about water quality? 



Traditional class learning 

Learning through Quest 
Atlantis (Taiga) 

Traditional class learning + 
Learning through Quest 
Atlantis (Taiga) 

TM1 (n=12) 

1 (8.33 %) 

8 (66.67 %) 

3 (25 %) 

TM2 (n=12) 

1 (8.33 %) 

7 (58.33 %) 

4 (33.33 %) 

TM3 (n=8) 

1 (12.5 %) 

7 ( 87.5 %) 

0 

Total (n=32) 

3 (9.38 %) 

22 ( 68.75 %) 

7(21.88%) 


In a similar question to that of the previous one, the students were asked about the method in which they 
could have enjoyed the most while learning about water quality. Surprisingly, most of the students of TM3, 5 
(62.5 %), expressed that they could have enjoyed most in the traditional class learning as is shown in Table 3. 
this could be because, as the researcher observed, the students of TM3 were very unsure about whether they 
are going in the right direction or not during the study. This created a sense of uncertainty in them. This could 
be because they are not used of learning outside the class, that too, from a video game or a virtual world. 

Most of the students of TM1, 8(61.54 %), and half of the students of TM2, 6 (50 %), thought that they 
could have enjoyed the most while learning through QA. one can infer from these results that it is necessary 
to hold simultaneous explanatory lessons in class to keep the confidence of the students in their learning 
process. 

Table 3. In which of the following method do you think you could have enjoyed the most about water quality? 



Traditional class learning 

Learning through Quest 
Atlantis (Taiga) 

Traditional class learning + 
Learning through Quest 
Atlantis (Taiga) 

TM1 (n=13) 

2 (15.38 %) 

8(61.54%) 

3 (23.08 %) 

TM2 (n=12) 

2 (16.67 %) 

6 (50 %) 

4 (33.33 %) 

TM3 (n=8) 

5 ( 62.5 %) 

2 (25 %) 

1 (12.5 %) 

Total (n=33) 

9 (27.27 %) 

16 ( 48.48 %) 

8 (24.24 %) 


3.2 About Likeness of QA 


The students were inquired if they had liked using QA during this research or not. About half of them, 16 
(48.48 %), agreed that QA is fantastic (Table 4). Another 10 students, 30.30 %, expressed that it was good to 
use QA. Thus, if we combine both who said it was fantastic and those who said it was good then most of the 
students, 26 (78.79 %), liked using QA. The number was again low for the TM3 group as only 3, 37.5 %, 
who agreed that using QA was a fantastic experience. 

Table 4. Did you like using Quest Atlantis? 



Yes, it was very 
fantastic 

Yes, it was good 

It was ok 

No, it was not 
good 

Not at all, it was 
really bad 

TM1 (n=13) 

7 ( 53.85 %) 

4 (30.78 %) 

1 (7.69 %) 

1 (7.69 %) 

0 

TM2 (n=12) 

6 (50 %) 

4 (33.33 %) 

2 (16.67 %) 

0 

0 

TM3 (n=8) 

3 (37.5 %) 

2 (25 %) 

1 (12.5 %) 

2 (25 %) 

0 

Total (n=33) 

16 ( 48.48 %) 

10 (30.30 %) 

4(12.12%) 

3 (9.09 %) 

0 


In another question, the students were asked if their school should continue to use QA or not. Most of the 
students, 21 (63.64 %), agreed that the school should continue to use QA (Table 5). The number was highest 
in the TM2 group where 10, 83.33 %, agreed with the statement and it was lowest in TM3 in which only 3, 
37.5 %, agreed. 


Table 5. Do you think that your school should continue to use Quest Atlantis? 



No, not at all. It is useless 

Yes, it is very good 
educational software 

May be. I am not sure 

TM1 (n=13) 

2 (15.38 %) 

8 (61.54 %) 

3 (23.08 %) 

TM2 (n=12) 

1 (8.33 %) 

10 ( 83.33 %) 

1 (8.33 %) 

TM3 (n=8) 

2 (25 %) 

3 (37.5 %) 

3 (37.5 %) 

Total (n=33) 

5 (15.15 %) 

21 ( 63.64 %) 

7 (21.21 %) 
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The students were also asked if they would be using QA at home or not and 41.67 %, 10, chose ‘yes, 
definitely' while 19.17 %, 7, chose ‘yes, sometimes’. It means most of the students, over 70 % of them, take 
QA as a legitimate virtual world for leisure as well. 

These results show that overall, with the exception of TM3, most of the students liked using QA and they 
think that it is enjoyable and schools should use it in future. However, the way it shall be used in schools 
shall be considered carefully as it can affect the satisfaction of the students. 


3.3 Features of QA that were Liked 


The students expressed their likeness for different features of QA in a question in which they had to specify 
their level of likeness for ten features which were: missions / quests; graphics; teacher guidance system; 
gaming elements; topic; story; answering system; learning process; customization of avatar; and interface. 28 
students answered this question. Most of the students (n=28), 18 (64.29 %) liked the missions/quests the 
most (Table 6). Graphics was liked by 17, 60.71 %, and teacher guidance system was liked byl6, 57.14 %, 
students. The most disliked was the interface as 12, 42.86 %, students said it was ok and about 21.43 % said 
that they did not liked it much. The interface of any game, and virtual worlds, should be simple enough 
(Kohler, 2012) but it seems it was not in the case of QA. The same feelings were conveyed to the researcher 
by the students and by the teachers many times during the research. One of the reasons could be that the 
students and teachers were using a virtual world for the first time. 

Table 6. How much did you like the following in Quest Atlantis? 



Very much 

It was O.K. 

Not much 

Not at all 

Missions/ Quests 

18 ( 64.29 %) 

9(32.14%) 

1 (3.57 %) 

0 

Graphics 

17 (60.71 %) 

7 (25 %) 

3 (10.71 %) 

1 (3.57 %) 

Teacher guidance system 

16(57.14%) 

10 (35.71 %) 

1 (3.57 %) 

1 (3.57 %) 

Gaming elements 

15 (53.57 %) 

11 (39.29 %) 

2(7.14%) 

0 

Topic 

13 (46.43 %) 

12 (42.86 %) 

3 (10.71 %) 

0 

Story 

13 (46.43 %) 

7 (25 %) 

2(7.14%) 

6(21.43%) 

Answering system 

12 (42.86 %) 

11 (39.29 %) 

2(7.14%) 

3 (10.71 %) 

Learning process 

12 (42.86 %) 

10(35.71 %) 

5 (17.86 %) 

1 (3.57 %) 

Customization of the avatar 

1 1 (39.29 %) 

1 1 (39.29 %) 

4 (14.29 %) 

2 (7.14 %) 

Interface 

10(35.71 %) 

12 (42.86 %) 

6(21.43 %) 

0 


3.4 Problems and Hindrances 


The problems that were mentioned in the survey can be largely divided into the ones due to the design of the 
research and the technical problems. The owner of the school, administration, teachers, students, 13 (39.39 
%), and parents, all of them expressed to the researcher that it was a wrong time to conduct this research as 
final exams of the students were near and the school nor their students wanted to divert their attention as 
promotion of the students to the next grade depends on that. 14 students, 42.42 %, also mentioned that 
working in pairs was problematic. The main reason for that, as was mentioned by students during the study, 
was that the students could use only one QA account to proceed. The student having the other account was 
feeling left behind and some students tried QA from home to cover the difference. Students and teachers also 
complained about the short session times as was also mentioned by Koh et al (2012: 56). Each session was 
about 40 minutes long as per guidelines from Atlantis Remixed t team but it proved to be short on many 
occasions. It is recommended that each session shall be at least 60 minutes long for non-native speakers of 
English. 

Most of the students, 16 (48.48 %), and teachers also complained about the slowness of the internet. 
Although, the internet had sufficient speed but the bandwidth proved to be lacking when all computers tried 
to access QA at the same time. This resulted in QA getting stuck many times or the answers not sent 
properly. The computers barely met the minimum requirements and had to be upgraded for this study which 
resulted in some spending for the school. The technical problems were also a cause of concern in Second Life 
(Woods, 2010: 105-108) and have to be sorted out to make the experience better. 
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4. CONCLUSION 

This was an important study for many reasons. Firstly, it was one of the few researches that have been 
conducted in developing nations on the use of virtual worlds in education especially at schools. Given the 
Pakistani context, the research can prove to be very beneficial for future studies and can serve as a starting 
point to incorporate digital educational technology in schools. To do this, one must take into consideration 
the timing of research, the nature of school system, local context, curriculum and culture. 

Mostly, students liked using virtual world in schools and would like to continue using it at their schools. 
However, the teaching methodology has to be chosen carefully to make sure the students are satisfied with 
the learning process and enjoy it too. 
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ABSTRACT 

This paper proposes a novel educational support system for technical experiments involving the production of practical 
electronic circuits for sound processing. To support circuit design and production, each student uses a computer 
during the experiments, and can learn circuit design, virtual circuit making, and real circuit making. In the 
proposed system, an analysis system that runs on the internet translates the composition of the circuit that is 
designed and constructed by the experimenter into the general circuit description language (SPICE). This circuit 
translation into SPICE enables the simulation of circuit behavior and indicates the presence of incorrect parts in 
the circuit. The analysis system also provides simulated sound, which is synthesized on the basis of circuit 
designed or constructed by an experimenter. Therefore, students can individually learn practical circuit making, 
behaviors of sound-processing circuits, and sound simulation. The proposed system is applicable to various circuit 
types and learning environments, such as computer-aided circuit design and the manufacture of both virtual and real 
circuits. The usefulness and effectiveness of the proposed system was evaluated by analyzing circuits made by five 
university students in an actual class. 


KEYWORDS 

Electronic circuit experiment, circuit making, sound signal processing, SPICE 


1. INTRODUCTION 

Knowledge of the design and production of electronic circuits and their practical applications is important 
for education involving technical experiments. Recently, several educational support systems have been 
developed to improve students’ understanding of electronic circuits and their circuit-making ability. 
Remote learning systems and virtual laboratories are useful for students lacking experimental 
facilities (Gurkan, 2008; Oliver, 2009). The simulation systems were developed for learning electronic 
circuits used in the field of engineering (Abramovitz, 2011; Fitch, 2011). However, these conventional 
systems cannot cope with the wide variety of circuits constructed by individual experimenters because they 
are based on all-purpose or ready-made learning tools and are applicable to only specific circuits within a 
subject area. Moreover, technological experiments in educational institutions have the following problems: 
(1) student-to-instructor ratios are usually very large and (2) mistakes made in an experimenter’s circuits 
can cause serious accidents such as electric shocks or fire. It is therefore difficult to ensure efficient and 
secure individual instruction. 

To overcome these disadvantages, I have proposed an educational support system for distance learning of 
experiments involving virtual circuit making (VCM) and real circuit making (RCM) (Takemura, 2011). The 
previous system (Takemura, 2012) was developed so that a student can learn circuit design and make 
practical applications of the produced circuit in an electric guitar (Fender Stratocaster). However, the 
graphical user interface (GUI) used in the preceding VCM systems is not sufficient for practical use. In this 
paper, a new VCM system including a more user-friendly GUI is proposed (cf., subsection 2.1). Moreover, 
this study improves the applicability of the system so that a student can learn practical circuits for sound 
signal processing. The proposed system is applicable to various educational modes and media (e.g., 
virtual laboratories and e-Learning) for practical electronic circuits and experiments including 
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practical circuit making, such as circuit design, VCM, and RCM (cf., subsection 2.3). The usefulness 
of the proposed system was evaluated by analyzing the circuits constructed by five students. 


2. METHODOLOGY 

Figure 1 illustrates the proposed educational support system for experiments involving the design and 
construction of sound processing circuits. This system consists of individual users’ computers and a remote 
analysis system. To support circuit design and production, each student uses a computer and transmits the 
circuit image to the analysis system that runs on the internet. The analysis system performs circuit 
recognition of the designed and constructed circuit using image processing techniques. The analysis system 
can automatically recognize the circuit construction, and it can also translate the structure of the circuit into a 
general circuit description language (SPICE) (Rabaey, 2012). This SPICE translation technique (Takemura, 
2011) can indicate how the circuit works, and it can identify incorrect parts that may exist in the circuit. This 
is an important step that aims to improve the efficiency and prevent the occurrence of accidents. The 
proposed system improves the circuit recognition technique for the VCM and RCM to make it applicable to 
educational support systems for the practical circuit making of a sound processing circuit. As shown in Fig. 
1, the analysis system can provide each experimenter with simulated sound, which was synthesized using the 
designed or constructed circuit. Therefore, individual experimenters can better understand the effects of 
sound processing achieved with their circuits. 



Figure 1 . Schematic of the proposed educational support system for experiments involving production of the sound- 
processing circuits. 


2.1 Educational Support System for Circuit Design and Virtual Circuit 
Making 

The VCM system is useful for students lacking the facilities or equipment for RCM (e.g., 
laboratories, circuit components, and measurement instruments). The previous VCM system (Takemura, 
2012) allows individual experimenters to use general graphics editors, which are usually provided in a user’s 
computer. Therefore, this system does not require software to be used exclusively. A virtual circuit image is 
transmitted to the analysis system along with information pertaining to the color coding of each drawn line. 
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which indicates connected circuit components. The analysis system can differentiate between circuit 
components and wiring on the basis of line colors, which are defined by the experimenter. However, the 
virtual circuit image based on the user-defined color lines is actually poor. Thus, the GUI of the preceding 
system is not user friendly. 

This paper improves the GUI of the VCM system so that connections of circuit components on a virtual 
circuit can be indicated by an experimenter by placing the virtual circuit components and characters 
representing their parameters (e.g., IC, resistance, and capacitance) on a template circuit-board (breadboard) 
image using the experimenter’s preferred graphic editor. Figure 2 shows examples of the virtual circuit 
components. Each experimenter can download these virtual circuit components and the template image from 
the analysis system. The experimenter indicates wirings in the virtual circuit by drawing colored lines on the 
template image using the graphical editor. The analysis system can perform circuit recognition using image 
processing techniques (pattern matching) between the image of a virtual circuit and the database of each 
circuit component in the server computer of the analysis system. The analysis system translates the virtual 
circuit into SPICE without errors because circuit recognition is based on data that is related to the input of the 
virtually constructed circuit. 



Figure 2. Circuit components for the VCM system: (a)-(d) virtual circuit components prepared for connection on the 
circuit board image (resistor, potentiometer, capacitor, and IC, respectively), and (e) circuit board image used to construct 

a virtual circuit. 


2.2 Educational Support System for Real Circuit Making 

An experimenter transmits the image of a constructed circuit to the analysis system after making the circuit. 
The analysis system can perform pattern matching between the virtually made circuit (drawn using the VCM 
system) and the circuit made using the RCM system. This allows the analysis system to automatically 
differentiate the layout of each device and the circuit wiring. In addition, the analysis system performs circuit 
recognition based on the training data obtained from this approximate differentiation and the circuit device 
database (Takemura, 2012). This pattern matching process between the virtual circuit and the real circuit 
improves the accuracy of circuit recognition and translation into SPICE, and decreases the computational 
cost of the circuit recognition and translation. 

2.3 Combination of the VCM and the RCM Systems 

The proposed system has an advantage that individual users can choose preferred modes (combination 
of VCM and RCM) and media depending on the required purpose or environment. For instance, the 
proposed system is applied to the following purposes: 

• A student lacking circuit components or equipment (measurement instruments) can virtually 
experience circuit making and measurement. 

• An experimenter can learn the behavior of the circuit constructed using the VCM or the RCM system on 
the basis of circuit translation into SPICE. Measurement instruments are not needed to check the circuit 
behavior. 

• An experimenter can construct the circuit of an electric guitar (Fender Stratocaster) and practical circuits 
for sound signal processing using the VCM or the RCM system depending on the purpose or 
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environment. Moreover, the experimenter can experience the simulated sound obtained from the 
constructed circuit without the need for audio equipment (e.g., audio amplifier and loudspeaker). 


3. RESULTS AND DISCUSSION 


The proposed system is evaluated by analyzing circuits, which were designed and constructed by five 
experimenters in an actual class at Tokyo University of Agriculture and Technology. Figure 3 shows an 
example of the results based on a practical sound processing circuit (low-pass filter) that was used to improve 
the sounds of an electric guitar. Figure 3(a) shows the diagram of the circuit to be made. Figure 3(b) shows 
the incorrect circuit of the sound processing circuit, which was drawn using the experimenter’s computer. 
The analysis system automatically indicated the incorrect parts on the circuit image using red broken lines. 
The experimenter then corrected the circuit accordingly. Based on the corrected circuit [shown in Fig. 3(c)], 
the analysis system recognized the circuit construction, and translated the circuit into SPICE. Figure 3(d) 
shows an example of the simulation results (frequency characteristics) obtained from the results of the SPICE 
translation. The analysis system provided this simulation result and simulated the sounds, and the 
experimenter could therefore learn the behavior of the circuit and the effect of sound processing (synthesized 
sound) on the basis of virtual circuit. Figure 3(e) shows the circuit constructed using the RCM system 
depending on the correct circuit, which was constructed virtually using the VCM system [Fig. 3(c)]. The 
results of the circuit simulation and sound processing obtained from the real circuit made using the RCM 
system were the same as those obtained from the virtual circuit, which was previously made using the VCM 
system. The user can then verify that the circuit made using the RCM system behaves accurately. 



Figure 3. Results of automatic circuit translation and simulation using the VCM and RCM systems: (a) diagram of a 
circuit to be made; (b) incorrect circuits made using the VCM system; (c) correct circuits made using the VCM system; 
(d) SPICE simulation based on the circuit translation of the circuit (e); and correct circuit constructed using the RCM 

system. 
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The proposed system was evaluated by analyzing the circuits made by five students at Tokyo University 
of Agriculture and Technology. The following positive responses, which pertain to the usefulness and 
efficiency of the proposed system, were obtained from all of the students: 

• The proposed systems are useful because it enables remote learning of topics such as circuit design 
and experiments involving practical circuit making for sound signal processing. 

• The proposed system is instructive because individual students can learn the practical circuits 
for sound signal processing, and also the behavior of circuits on the basis of SPICE and 
simulated sounds. 

• The GUI for VCM is user-friendly and useful. 

However, there were also a few technical disadvantages and suggestions for improvement: 

• A remote educational system to learn both analog and digital circuits for signal processing is 

expected. 

• The size of the experimenter’s system is expected to be reduced. 


4. CONCLUSION 

This paper proposed an educational support system for experiments, which are performed in different 
environments such as computer-aided circuit design, VCM, and RCM. The system can teach methods related 
to the production of basic circuits for processing sound signals. The usefulness and effectiveness of the 
system were verified by analyzing electronic circuits made by five students in an actual university class. The 
following steps are necessary to practically realize the proposed system: 

• The usefulness of the proposed system must be first validated by a larger number of experimenters. 

• There is a need for an educational support system, which can be used by users to learn various 
techniques and practical circuit making (e.g., larger scale circuits). 
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ABSTRACT 

Distance Learning supported by the WEB is a reality which is growing fast and, like any technological or empirical 
innovation, it reveals positive and negative aspects. An important aspect is in relation to the monitoring of the activities 
done by the students since an accurate online assessment of the knowledge acquired is an open and, therefore, worrying 
subject. A way of minimizing such situation is by using technology to monitor the general performance of the student in 
the virtual learning environment. This work invests on the construction of the OAEditor framework which allows 
teachers to create learning objects in the form of educational hyper documents representing conceptual graphs that can 
adapt to the interaction of the students according to their performance, besides enabling a pedagogical monitoring based 
on self-evaluative questionnaires which are automatically generated, pondering the navigation and performance obtained. 
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1. INTRODUCTION 

This paper describes a framework to edit the Learning Objects (LOs) able to adapt to single characteristics of 
the students in the process of Distance Learning (DL). Most of the LOs produced nowadays do not provide 
feedback information supporting their efficiency in the teaching process, registering, for example, if the 
student is interacting in the right way with the object and if it is reaching the planned objectives Moodle 
(2012), BlackBoard (2012), Amorim (2005). Studies warn that the learning process is made easier when the 
teaching methods are in agreement with the learning style of the student Bohlen (1993, pp. 280-281), Graf 
(2008). 

The solution proposed here is the OAEditor framework integrated to the AVA Moodle that performs the 
following functions: 

• It enables the creation of learning objects in the format of educational hyper documents representing 
conceptual graphs; 

• It allows the interaction between students and these learning objects; 

• It generates logs in the interaction student-object; 

• It stores a question bank; 

• It automatically generates questionnaires to evaluate the level of learning of the students by choosing 
questions from a question bank that involve their navigation history and their development; 

• It generates data reports of the logs and the students’ assessments. 

Moodle, the most popular Distance Learning Environment (DLE) among the eLearningGuild users Davis 
(2009), allows the aggregation of new blocks of functions - plugins - which broadens its functionalities. The 
OAEditor was designed as a plugin of Moodle and it was developed with the PHP language. 

1.1 Structured Hyperdocuments as Conceptual Graphs 

According to Bush (1945), the human mind operates by associations, following an intricate web of 
representations. For that reason there is a necessity of breaking with the conventional hierarchical structure 
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that we find in books, trying to adapt the didactic material to the nature of the human reasoning. The 
hypertexts allow the non-linear writing/reading where one can edit and manage the texts in a distributed 
environment. With the use of this facility the users have the possibility of interacting, creating new texts, 
introducing new documents or modifying the documents which are already there. The OAEditor uses this 
process in the creation of the LOs in which the professor creates fragments of information and also the 
connections among these fragments implementing Conceptual Graphs (CG), a concept proposed by Sowa 
(2004) to a graphic representation of knowledge, where the nodes represent the concepts and the edges 
represent the links among the concepts. 


1.2 Applications Sensitive to Context 

Medeiros (2010, pp. 15-16) presents a summary of some structures in the adaptive environments in relation 
to the following characteristics: data acquisition, storage, processing, characteristics of context use, patterns 
of the adopted content and communication infrastructure. The OAEditor has some characteristics of these 
structures, composing a new proposal, as shown in Table 1: 


Table 1. Synthesis of structures and synthesis of the OAEditor 


Synthesis of adaptive structures by Medeiros (2010, pp. 15-16) 

Synthesis of the 
OAEditor 


Semantic 
Learning Space 

CTRL 

GlobalEdu 

Adaptation of 
Learning Scenarios 

Acquisition 

Physical and 
Virtual sensors 

Interactions 
with users, 
Model of the 
student and 
Curriculum 

Learner Profile, 
Elements on social 
and Physical Context 

Interactions with the 
user, Model of the 
student 

Interactions with the 
user. Model of the 
student 

Storage 

Ontologies of 
content, context 
and domain 

XML 

Ontologies 

Objects Repository 

XML 

Processing 

First-order logic. 
Bayesian nets, 
Fuzzy logic and 
probabilistic logic 

Logic Rules 
of IF THEN 

Inference on the base 
of knowledge using 
the behavior rules 
WHEN IF THEN 

Machine learning 
algorithm- 
(KNearestNeighbor) 

Logic Rules IF 
THEN 

Use of context 

Recommendation 
of contents 

Intelligent 

tutoring 

Recommendation of 
learning objects and 
the support to the 
interaction of the 
learner 

Adaptation of 
learning scenarios 

Recommendations of 
learning objects and 
support to the 
learner's interaction 

Patterns of content 

LOM, SCORM, 
IMS Learning 
Design, OKI, 
Ariadne and Dublin 
Core 

Not applicable 

SCORM 

LOM (Learning 
objects) and IMS 
Learning Design 
(Learning Scenarios) 

Conceptual graphs 

Infrastructure of 
communication 

Centralized 

Distributed 

Centralized 

Centralized 

Centralized 


The great difference of the OAEditor is the proposal of structuring the hypertexts as conceptual graphs, 
providing the framework a big range of possibilities of implementation. 


2. ARCHITECTURE/CHARACTERISTICS OF THE OAEDITOR 

The Fig. 1 exposes the number of use cases in the specification of the OAEditor: maintaining learning 
objects, maintaining the question bank, logs, interacting with the learning objects, evaluating the learning 
process and Data evaluation. 
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2.1 Use Case: Maintaining Learning Objects - Available to Teachers/Tutors 

In the OAEditor the text is typed in a similar way to the editor of a regular text. One of the most important 
characteristics of the OAEditor is that it is adaptable to the student’s profile, and its concepts are available in 
three levels: ‘synthetic’, ‘detailed’ and ‘learn more’, where each level presents the same concept, however, it 
is presented in different ways in a growing sequence of detailing. 

2.2 Interacting with the Learning Objects - Available to Students 

The OAEditor uses the authentication model of Moodle to get all the information about the user who is using 
the system, information such as login, profile, relationship discipline-student, section, and so on. Two users’ 
profiles are considered: Teacher/Tutor and Student. The teachers can create/update/exclude LOs, feed the 
question bank and follow students individually. The students will be able to navigate through the LOs created 
by the teacher, answer questionnaires and see their personal records. 

2.3 Logs 

The interaction of the students with the LOs produces registers related to their navigation on the LO. With 
these logs many questions can be explored: 

• Did the student really choose a satisfactory knowledge path? 

• Which concepts were not chosen to be navigated? 

• A certain concept was not chosen because the student already mastered it or because it was a lack of 
attention? 

The great gain that it generates to the teacher is the possibility of knowing the individual profile of the 
students offering them more precise feedback. 

2.4 Maintaining the Question Bank - Exclusive to Teachers 

A question bank with good quality questions related to all concepts present in the LO is necessary to make 
the generation of questionnaires possible. The defined architecture suggests multiple choice questions with 
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five alternatives and only one possible answer, where each question is related to a specific concept present on 
the LO. 

2.5 Evaluating the Learning Process 

This evaluation is done when the student requests it. Each questionnaire has ten questions which are chosen 
in a way to have all the concepts of the lesson, favouring the concepts which were not navigated and the ones 
which obtained the lower results quantitatively. With this, the OAEditor breaks the conventional paradigm 
that the same questionnaire is applied to all students. When it is answered and submitted, the OAEditor sends 
the results back to the student's assessment, showing the mark and correct answers. 

2.6 Data Evaluation 

The pedagogical assessment will take place through an individual monitoring with each student. With the 
OAEditor the teacher has access to a monitoring area of the students attending the discipline where the way 
each student interacted of the learning object is shown. It is possible to visualize how many times each 
student accessed each concept, and there is also a table showing the number of questions about each concept 
and the number of right and wrong answers to each question together with all the tests taken by the student. 
All the tests can also be visualized. 

On the page for monitoring the students you can see the tree of concepts of the unit on the left, and it 
shows how many times the student accessed each concept. There is also a table informing the number of 
questions related to each concept in all the tests done by the student as well as the number of right and wrong 
answers. And, lastly, the teacher can access the tests taken by the students. 


3. CONCLUSION 

The OAEditor framework implemented enables the creation of learning objects to the teacher in a transparent 
way. The teacher does not need to have specific knowledge in programming to create a high quality LO. The 
OAEditor transforms the act of creating an LO into a simple typing activity. From that moment, the system 
works in an automatic way, keeping the navigation logs of the students, automatically generating tests based 
on the navigation of the tests already solved, and also generating analytical reports to the teacher. The quality 
of the learning object created is directly associated with the expression of the conceptual graph in terms of 
completeness/coverage of concepts and established relationship among the concepts. 


4. FUTURE WORKS 

The main challenge that can be the motivation to a development of future studies is the incorporation 

resources of the artificial intelligence (AI) to the OAEditor. The use of the AI would help the development of 

a structure which permits: 

• The programming of the LOs to suggest individual contents, based on cases, increasing the quantity of 
parameters to be considered; 

• The creation of a mechanism of intelligent tutoring where the system itself can give the tutoring by 
suggesting contents, questions, external material among other things registering cases of success. Besides, 
it suggests paths the students should take based on these cases; 

• The increase in the number of criteria which compose the generation of assessment questionnaires and 
establishing metrics able to detect if the questionnaires were enough to an evaluation criteria; 

• The improvement of the reports in order to obtain more consistent data to take the decision, that is, 
establish more precise filters to build the reports. 

Other perceptions that can be applied latter on are: 

• Making the system become collaborative so the students can insert contents in the LOs that could be 
assessed by the teacher and made available to the community, making it something similar to a wiki. 
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• Using the data mining techniques to obtain a better benefit front the information related to the students’ 
logs; 

• Making the system, without the intervention of the teacher, suggesting materials for studying, questions to 
solve, and so on; 

• Identifying if there are patterns or tendencies in the navigation and answers of the students. 
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ABSTRACT 

This article presents the proposal of a social network site SocialNetLab that belongs to the Department of Computing - 
Federal University of Sergipe and which aims to locate and notify users of a nearby friend independently of the location 
technology available in the equipment through dynamic Web Service; to serve as a laboratory for research in social 
networks and to promote collaborative learning and integration among students, teachers and researchers, providing the 
institution’s services and educational resources in full time. 


KEYWORDS 

Mobile Social Network, Social Network for Education. Dynamic Web Service, Technology Location. 


1. INTRODUCTION 

The increasing use of mobile devices, especially cellular and its derivatives such as smartphones, have 
favored the expansion of available services through the Internet. The site of the ComputerWord published in 
February of 2011 that more than half of computing devices sold in the coming years will not be computers 
but smartphone, tablets and netbooks (Fonseca, 2011). 

According to the magazine Epoca, published in May of 2010, the world-wide average of virtual friends is 
195 persons per user, when in Brazil is 365, being that more than 80% of the Internet users are registered in 
at least one social network, classifying Brazil as the country most sociable of the world (Ferrari, 2010). 

After these considerations we realize the importance of conducting research on social networks. The 
objective of this article is to present the mobile social network SocialNetLab (Social Network Laboratory) 
that will serve of laboratory for researches on social networks, will allow users to find friends and be notified 
of the proximity of them according to configuration of distance defined by the user, through a movable 
device independent of the technology of location available in the equipment, may also provide tools to the 
academic community. Besides this introduction, the article is divided in more four sections described below. 
In section 2 we conceptualize and describe some social network sites and present the advantages of 
SocialNetLab in relation to other social network sites. In section 3 we present the social network site 
SocialNetLab as well as the functionalities and applications that will make up the site. In Section 4 we show 
the progress of implementation of SocialNetLab. In section 5 we present the final considerations and the 
future works. 


2. MOBILE SOCIAL NETWORKS 

The social network sites or SNS are web-based services that allow individuals to: 1) construct a public or 
semi-public profile within a bounded system; 2) articulate a list of other users with whom they share a 
connection; and 3) view and traverse their list of connections and those made by others within the system 
(Boyd. 2011). 
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There are online social networks (Web-based), mobile social networks and hybrid social networks. A 
hybrid social network is initially being developed as online, and migrates or extends to mobile platforms 
later (Tong, 2008). What differentiate an online social network of a mobile social network are the 
technological facilities of the new mobile devices combined with the features of the social networks online. 

The idea of implementing a mobile social network own arose because of the non-existence of a mobile 
social network non-commercial and open source to accomplish the researches of the institution. 


3. SOCIALNETLAB 


The SocialNetLab is an SNS that will allow a user to have access to educational tools and also enable the 
user to locate and be notified of the proximity of a friend regardless of location technology available in they 
equipment and according with the configuration of distance established by the user, making the SNS more 
attractive. 

To do the location of the mobile device we will use initially the API of Geolocation that is already 
incorporated to the HTML 5 (Meyer, 2011). This API is a hybrid positioning technology proposed by the 
working group Ubiquitous Web Applications of the W3C (World Wide Web) consortium. The idea is that 
getting the coordinates (latitude and longitude) will occurs when the user accesses the SocialNetLab site. 

As the Geolocation API serves only for four types of location technologies, we realized the need to 
implement another solution that will serve the higher number of location technologies available in the 
market. In an attempt to guarantee the independence of location technology in mobile device is that the idea 
appeared of using the Web Service Dynamic (Murray et al, 2006), so that the system will go automatically 
search for the appropriate service for the technology of available location in the equipment. In the figure 1 
we can see the architecture of the SocialNetLab SNS. 
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Figure 1 . Architecture of the SocialNetLab 

In this environment will be provided educational tools that will serve as manager of educational contents, 
will function as remote laboratory in electronics, will serve as a judge online and as a tool for learning 
through play. These tools are detailed below. Besides these features, SocialNetLab will also serve as a 
laboratory for research on social networks. 

The figure 2 shows a prototype of the website's homepage of the SocialNetLab. The site of the 
SocialNetLab will have the Tools’s link in which will be available the tools directed to education as the 
LEW (Laboratory Engineering Web), the ERLab (Electronic Remote Laboratory), the ProgWeb and the 
JOnline. These tools are already implemented and will be incorporated in future to the SocialNetLab. 

The LEW is a tool that is already in use through a portal and which provides an environment for the 
managing information related to teaching, research and extension. (Ribeiro et al, 2011). The lOnline is a 
online judge which seeks to support the programming learning (Santos et al, 2011). The ProgWeb is a tool 
whose purpose is to build an environment for teaching programming through games using the programming 
language Python. (Ribeiro et al, 2011). 
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Figure 2. Prototype of the initial web page of SocialNetLab 

The ERLab is a middleware for remote access laboratories (Ribeiro et al, 2012). It allows users to access 
various resources (oscilloscopes, signal generators, multimeters, reprogrammable hardware, among others) at 
a distance, through Internet. (Feistauer et al, 2010). In figure 3 we can see the homepage of the ERLab and 
an oscilloscope in functioning through the ERLab. 
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Figure 3. ERLab and oscilloscope 


4. PRELIMINARY RESULTS 


The social network SocialNetLab is being implemented and was divided into three scenarios as shown in 
figure 4. 


Scenario, 
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K 



Figure 4. Implementation Scenarios of the SocialNetLab 
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The scenario 02 is in the final phase of the implementation and is being developed for any device (mobile 
or desktop) with Internet access can run the software, find friends and be located, without this being 
necessary to install and run any application at the client. 

The ultimate objective is to implement successfully the scenario 03, allowing, in addition to mobile 
devices with Android, all other devices with Internet access have access to dynamic localization. For this we 
are studying and researching the best solution to integrate the Dynamic Web Service application (scenario 
01) with the API Geolocation application (scenario 02). Such integration could be done by connection A, 
where access to the Web service would be made while accessing the browser, or through connection B, 
where access would be made when the application accessing the server. 


5. FINAL CONSIDERATIONS 

The mobile social network SocialNetLab is functioning with the location system via Web Service for mobile 
devices that use Android. As for the other types of devices (mobile or desktop) the application is in the final 
phase of development, so that the main objective of the implementation is the integration between the 
solutions of location Web Service dynamic and API Geolocation making the application always available to 
the user and can be run on any device with Internet access. Finally we will take the integration of educational 
tools, the tests and the analysis of results found. 

As future work we will be integrating all the tools that will compose the educational environment, as well 
as adding new features that add value to the SNS. Issues of privacy and security will be also covered in more 
detail later. 
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ABSTRACT 

This paper presents the online cognitive and algorithm tests, which were developed in order to determine if certain 
cognitive factors and fundamental algorithms correlate with the performance of students in their introductory computer 
science course. The tests were implemented among Management Information Systems majors from the Philippines and 
among the students of the Information Systems track in a technology high school in Japan. The cognitive factor. Spatial 
Scanning, correlated significantly with the performance of both student groups in introductory computer science. Among 
the algorithms, Maze Tracing correlated with the university students’ final grade and Binary Search with the midterm 
exam result of the high school students. The online tests also showed significant correlations among Searching, Sorting 
and Maze Tracing algorithms. The obtained correlations may be considered in creating assessment tools and in designing 
learning strategies for introductory computer science courses for both university and high school levels. 


KEYWORDS 
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1. INTRODUCTION 

Researchers, particularly in the field of Computer Science Education, have endeavored to find ways on how 
to address the high drop-out rate and the poor performance of students in computing. These studies (Irons D., 
1982, Rountree, N., et.al., 2004, Simon, et.al, 2006) attempt to identify the factors that determine success in 
computing and measure aptitude in programming. In relation to this, the importance of mastering 
programming early was emphasized by the ACM Computing Curricula 2001 Final Report which proposes 
programming as a prerequisite for many advanced courses in computer science (ACM, 2001). It is also 
important to find ways to improve computer science for high school as it is now being considered as an 
essential subject along with the traditional STEM (science, technology, engineering and mathematics) (CSTA, 
2005). 

The main goal of this research is to determine which among known cognitive factors and fundamental 
algorithms correlate with the actual student performance in introductory computer science through online 
tests. The tests can be used as reference for designing assessment tools for both university and high school 
students who plan to study computer science and other technology and computing-related degrees. The 
correlation results may also be used in determining possible ways to help students succeed in introductory 
computer science, specifically in programming. 


2. METHODOLOGY 

The first phase of this research covered the initial development of the online cognitive and algorithm tests 
and their preliminary implementation among sixty-one (61) second year Management Information Systems 
(MIS) majors at the Ateneo de Manila University. These participants have taken up Introduction to 
Computing I (CS 21A) and were enrolled in Data Structures and Algorithms (CS 21B) when they took the 
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tests in May 2007. The second phase involved the translation of test instructions to Japanese and their 
implementation in September, 2011 among forty (40) students from the Information Systems track of 
TokyoTech High School. At the time of the implementation of the tests, these students have already finished 
introductory programming but have not yet studied the fundamental algorithms considered in this research. 

2.1 The Cognitive Factors Tests 

Two sets of online tests were developed for the purposes of this research. One set was based on known 
cognitive factors from the Kit of Factor-Referenced Cognitive Tests by the Educational Testing Services 
(ETS), a US-based educational institution. Each cognitive factor has corresponding paper tests. The first 
cognitive test, Object-Number for the cognitive factor Associative Memory, assesses the ability of the student 
to learn combinations of words and numbers. The second is Letter Sets Test for Induction, which aims to find 
the rule that makes the letter sets similar and to identity which set does not fit the rule. The third. 
Diagramming Relationships test for Logical Reasoning, entails finding the relationships among things in a 
group, represented by overlapping circles. The last is Map Planning test for Spatial Scanning, the goal of 
which is to find the shortest route between two points in a city map the shortest possible time and according 
to a set number of rules (Ekstrom, et.al, 1976). A sample map for the Map Planning Test is shown in Figure 1 
below. Previous researches have shown that the aforementioned factors are related to programming aptitude 
(Irons, 1982; Scanlan, 1988). For the Japanese high school students. Diagramming Relationships Test was 
not included because this particular test uses English words and the authors have no license to modify and 
translate actual exam contents. 
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Sample routes 
A to Z passes building 1 
E to S passes building 5 
P to J passes building 3 


Figure 1. Sample map and routes in Map Planning Test 


2.2 The Algorithm Tests 


Fundamental algorithms, commonly taught in introductory computer science were designed as games that 
can simulate how the algorithms work. The first set of games is Card Searching which includes Linear 
Search and Binary Search. The goal is to locate a certain number from among ten (10) numbered cards. The 
next is Sorting games, where the goal is to simulate Insertion Sort and Selection Sort while sorting ten 
numbered cards in ascending order. The third game is Maze Tracing which attempts to simulate how breadth- 
first and depth-first search algorithms work. The last algorithm. Tower of Hanoi, checks the student's ability 
to think recursively. It was not implemented among the high school students because recursion is not covered 
in their course syllabus. Figure 2 shows the screen components of Card Sorting. 
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Figure 2. Unsorted cards and data boxes in Card Sorting 
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3. DATA ANALYSIS 

The scores from the online cognitive factors tests were collected. For the algorithm tests, data based on the 
execution of the games were used to compute the test scores. The test results were then correlated with the 
actual performance of the students in their introductory computer science course, the grades in CS 21A of the 
university students and the scores in the mid-term exam on algorithms of the high school students. Data 
analysis was done using Pearson's R correlation function in SPSS. 

3.1 Correlations of Online Tests with Performance in Introductory Computer 
Science 

Among the cognitive factors. Spatial Scanning correlated with the performance of the university students in 
CS21A (r=.331, R=.1095, sig=.010). This finding is similar to the study of Irons where this factor is one of 
those considered to be most relevant to programming aptitude. Spatial Scanning is associated with debugging 
skills as it entails finding “particular configurations representing a pathway through the field”. This is 
considered as a more psychologically relevant factor in predicting programming skill and is greatly related 
with debugging ability (Irons, 1982). Spatial Scanning also correlated with the midterm exam results of the 
high school students specifically with their score in the Binary Search problem (1-.383, R=.1466, sig=.015). 
This can be related with the ability to apply Binary Search algorithm where a certain scheme is employed for 
locating values in an array. Among the algorithms, only Maze Tracing correlated significantly with the 
performance of the university students in their introductory computer science class, CS21A (1-.310, R=.0961, 
sig=.017). In the case of the high school students, the result of the online test on Binary Search correlated 
significantly with the score in the Binary Search problem in the midterm exam (r=.352, R=.1239=, sig=.026). 
This particular result implies that it is more substantial to find the correlations between the online tests with 
the performance in the actual application of the algorithms rather than with the final grade. 

3.2 Correlation Results among Algorithms 

The results of the online algorithm tests were also correlated against each other as shown in Table 1. Two 
significant correlation results are similar for both university and high school participants. Linear Search with 
Binary Search and Insertion Sort with Selection Sort. Linear Search and Binary Search tests are expected to 
correlate because both are search algorithms and have the same objective, that of locating a value from a 
given set of cards. The same applies to the correlation between Insertion Sort and Selection Sort. 


Table 1. Correlation results among the algorithm tests 


Algorithms 

Correlation values ! 

University students 

High school students 

Linear Search with Binary Search 

1-.555, R=.3080, sig=.000 

r=.859 R=.7378, sig=.000 

Insertion Sort with Selection Sort 

1-.517, R=,2672, sig=.002 

r=,511, R=.2611,sig=.001 

Binary Search with Maze Tracing 

1--.314, R=.0985, sig=.023 

r=.465, R=. 2162,sig=.002 

Binary Search with Insertion Sort 

1-.501, R=.2510, sig=.003 


Binary Search with Selection Sort 

r=.375, R=.1406, sig=.011 


Maze Tracing with Tower of Hanoi 

1-.345, R=. 1 190,sig=.007 


Maze Tracing with Linear Search 


r=.390, R=.1521, sig=.013 


There are a number of differences, though, between the test results of the university students and high 
school students. For the university students. Binary Search and Maze Tracing are inversely correlated. These 
students who already knew Binary Search could have implemented the algorithm’s strategy in the 
corresponding online test but tried free search in Maze Tracing. However, for the high school students. 
Binary Search and Maze Tracing have positive correlations. This may be explained by the fact that searching 
is naturally applied in maze traversal. Both processes entail some movement from one point to another. 
Another possible reason is that the high school students have not yet studied Binary Search at the time of the 
online test implementation. Thus, they could have applied free searching in both algorithm tests, that is, not 
following a learned method. The same reason may be applied with the positive correlation between Maze 
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Tracing and Linear Search. Binary Search further correlated with Insertion and Selection Sort and Maze 
Tracing with Tower of Hanoi for the online test results of the university students but not for the high 
school students. These other correlations may be again due to the fact that the university students have 
already studied algorithms by the time of online test implementation. Their approach in executing algorithms 
is more unified and structured. Generally, the correlations may also be due to the natural connection between 
the concerned algorithms. Search strategies are naturally employed when sorting or tracing a maze. 


4. CONCLUSION 

Online cognitive and algorithm tests were developed and implemented among university students who are 
majoring in information systems and high school students who are in a similar course track. The aim was to 
find out correlations among the known cognitive factors and fundamental algorithms with the performance of 
the students in their introductory computer science course. 

A number of significant correlations were obtained and similarities as well as differences in the 
correlations between the two groups of participants were observed. The correlation results indicate 
consistency in the students’ performance in the online test and demonstrate certain reliability among the 
developed algorithm tests. Moreover, the obtained results show close connections among the cognitive factor 
Spatial Scanning and Maze Tracing and Binary Search, making them possible factors to be considered in 
learning computing fundamentals. 

In summary, the online cognitive factors and algorithm tests that correlated with the performance of the 
students either with the final grade or exam score may then be considered in designing tools in assessing 
student ability to succeed in an introductory computer science course. The said cognitive factors and 
algorithms may also be taken into account in the development of a course outline for introductory computer 
science and in preparing actual problem exercises for the students. 

Future work for this research includes considering other cognitive factors and algorithms for the online 
tests. The other data on the online test performances that were not used in scoring the algorithm tests may 
also be used in constructing other formulas for scoring the tests. These data may then be used to further 
study particular aspects with which the students implement algorithms and determine other factors that can 
assess aptitude in introductory computer science courses. 
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ABSTRACT 

In this paper different type of mathematical explanations are presented in relation to the mathematical problem of 
probabilities Monty Hall (card version) and the computational tool Latent Semantic Analyses (LSA) is used. At the 
moment the results in the literature about this computational tool to study texts show that this technique is appropriate for 
the case of expository and narrative texts, but there is not evidence with mathematical texts. The technique could help us 
to identify which are the better explanations and if this is relevant to explain correct responses. 
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1. INTRODUCTION: DIFFERENT EXPLANATIONS FOR THE MHD 

In this paper different type of mathematical explanations are presented in relation to the mathematical 
problem of probabilities Monty Hall dilemma (MHD) (card version, see Tubati and Alonso 2003). We will 
see that there are several ways of arriving at the correct response to the dilemma without resorting to the 
application of the theorem of Bayes. We will explore these explanations in order to introduce the 
explanations provided by the participants. 

A simple and completely correct explanation for the participants’ continued belief in the probabilities 
after the elimination appeals to the invariability of the probabilities of the initial sets. The elimination is not 
informative from the point of view of the probabilities of the initial sets: since the beginning we have known 
that at least 1 of the 2 cards held by the informant is different from the ace; the elimination is informative 
from the point of view of the individual probabilities of the cards (since the cards of the informant change 
from having equal probabilities of 1/3 to having probabilities of 0 and 2/3), but these do not affect the 
probabilities of the initial sets. 

The illusory response and a possible explanation 

The initial probability of each card being correct is the same, 1/3. Once the informant has showed one of 
his cards (situation of elimination), we tend to make the illusory inference that “if there are only two cards, 
the probability of each of them of being the ace is the same (1/2)”, when the probabilities are in fact the same 
as they were at the beginning: for the decision-maker, p (ace) = 1/3, and for the informant, p (ace) = 2/3. 

A possible explanation for such an illusory response could draw from a heuristic of similarity, where in 
situations in which there are two alternatives, we think the probabilities are equal. 

Necessary and sufficient information for the correct resolution of the dilemma 

It is important to realise that the key point in the explanation of the dilemma is the situation of 
elimination. Any of the factors that facilitate the resolution of the dilemma in fact facilitate the 
comprehension of the implications of the situation of elimination. It is at this moment that it is necessary to 
bear in mind that the performance of the informant is conditioned, and this is relevant to their response. 

Let us consider, furthermore, that to overcome the strong illusory inference, it is not enough only to 
consider the variety of possible cases of performance of the informant (that is to say, to think that “if John 
has the ace and the 7, he will always show the 7, whereas if he has the 7 and the 8 he will show half of the 
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time the 7 and half of the time the 8), but it is necessary also to consider the globality (or generic sum) of 
these and to compare the number of times in which there is the ace and the 7 and the number of times in 
which there is the 7 and the 8. For in the case where the informant has the 7, for example, the reasoning 
would be as follows: “Every 3n/2 times that he shows 7, n will be with the ace and n/2 with 8.”, and 
specifying for a natural number n=2, “Every 3 times that he shows 7, 2 of them will be with the ace and 1 
with the 8.” Therefore, if the informant shows the 7, it will be with the ace more times than with the 8; this 
reasoning have been explored in several questionnaires (see, for example, Tubau, 2008). 


2. EXPERIMENT 

A version of the problem was designed (see Appendix for details of the game) and the experiment was 
separated among 3 conditions; they were administered at different consecutive stages. 

The first condition was a control condition to be compared with other versions in which an explanation is 
asked. Condition A included a “Help-sentence” that shows to them the ways in which the informant might 
perform (following the line suggested by Johnson-Laird, Legrenzi, Girotto, Legrenzi, & Caverni (1999)) that 
was also included in the other versions. 

In the conditions B and C, a “Double question” was designed in order to help participants to answer 
correctly the dilemma that asked them for an explanation (justification) for their responses. The aim of this 
experiment was to see the efficacy of self-explaining the problem and to see, in condition C, if the same 
correct explanation given by the participants in condition B would be enough to correctly understand the 
problem 1 . 

2.1 Method 

Material. See Annex of Maja, 2009. In condition B, question “(a)” will always be correctly answered if the 
statement was correctly comprehended. Question “(b)”, which in fact already implies the illusory inference, 
is the one that theoretically could imply serious problems; more specifically, it could be similarly difficult to 
the two questions (Decision and Probability) that finally were considered in order to evaluate the correct 
resolution of the dilemma. Participants. 22 students of the University of Barcelona participated in exchange 
for course credit (one of the participants was eliminated due to previous familiarity with the problem); in 
condition B, 52 students; in condition C, 58 students of the University of Balearic Islands participated. 

2.2 Results 

The results show a surprisingly high percentage of switches (correct responses) in the first illusory question 
and a strong decrease in the later questions, despite all of them inviting the same illusory inference. 

As was expected, in the Decision question (in which the participant was only given the option of 
switching or sticking with the initially selected card), better results than in the Probability question (in which 
the exact probabilities were also asked) were found in both the B and C conditions (following the tendency 
that there is more difficulty expressing the answers by exact numbers rather than relative probabilities). 

The difference between the questions of condition A and condition B was significant in the Decision 
question; it was not significant in the Probability question. In the condition B, by seeing the difference 
between questions, significant differences comparing the Decision question and the Probability question were 
also found (45% vs. 20%; x 2 (1, N=102) = 7.57, p<.01). Therefore, the fact of giving explanations seem to 
especially helpful for the correct comprehension o the problem. 


1 In Condition C, instead of the Help sentence, the complete explanation given for participants in condition B was included: i.e., the 

correct response, followed by the connective ‘because’, and followed by the explanation provided by most of the participants in 
condition A. 
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2.3 Analyses of Justifications 



Figure 1. Type of justification and correct responses (%) in the Decision and Probability questions for Condition B and C 

Comparing the justifications of the conditions B (in which the justification ‘C’ was the most common one) 
and C, there was an increase in the condition C of the type of justifications H' (not being significant). There 
was a significant difference between the type of justification and correct responses to the ‘Decision' and 
‘Probability’ questions, being the correctness of both the Decision and the Probability questions more 
predictable from the type of justifications “S” (perhaps more creative). 


Table 1. Classification of the justifications of the participants (conditions B and C) 


Types of justification 2 

Examples of justifications (of the 
participants) 

LSA 

coherence 3 

LSA (with 
implicit 
inference) 
coherence 

Difference 
between 
LSA_i and 
LSA 

C: 

Repetition of the 
consideration of the Cases of 
the Help-sentence 

“It’s more probable that John shows 
the ace because if he has the ace and 
the 7 he always shows the 7 [and if he 
has the 7 and the 8 he only shows it 
half the time]” 

0.391 

0.438 

0.047 

N: 

Consideration (explicit) of a 
consecutive Number of trials 

Long version exemplified with 6 cases 
“It’s more probable that John shows 
the ace when he has the ace and 7 

0.47 

0.491 

0.021 


because if they play several times and 
always show the 7 it would be that 
hides the ace.” 

0.295 

0.34 

0.045 

E: 

Double sense ( Equivalency ) 
of the relationship between 
the cards that John has and 
the card that he shows 

“If John shows the 7 it’s more 
probable that he has the ace because if 
John has the ace and the 7 it’s more 
probable that shows the 7.” 

0.36 

0.412 

0.052 

S: 

Invariability of the 
probabilities of the initial Sets 

“If John shows the 7 it’s more 
probable that he has the ace because 
John always has 2 cards” 

0.311 

0.33 

0.019 

H: 

John Hides the ace. 

“If John shows the 7 it’s more 
probable that he has the ace because 
the ace, because he is forced to hide 
it.” 

“If John shows the 7 it’s more 
probable that he has the ace because he 
hides the 8 only half of the time 
whereas he always hides the ace.” 

0.284 

0.333 

0.049 


0.29 

0.339 

0.049 


2 Considering only the correct responses Cine’ expresses 'Incorrect Responses’) 

1 1t is considered the value LSAassa 'LSA, Sentence to Sentence, adjacent, mean', the one that show more differences in the results. 
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2.4 Analyses with LSA 

There are several authors (for example, Louwerse, 2010) that hold that LSA is not only a computational tool 
for text analyses but also it has also implications in order to understand the process of cognition. One reason 
to argue this would be that when the computational technique LSA estimates that the coherence of a sentence 
is high, it says that on the basis of a mechanism of semantic activation similar to a connectionist one in which 
the activation (or coherence) of a sentence happens in relation to the activation of another (a mechanism of 
co-ocurrence) (as developed in Kintsch 1988, 1998, for example). 

Therefore, LSA could be used in order to study these justifications (explanations) and to see if a specific 
implicit inference (T) would be in the mind of some of the participants that correctly answered the problem 
until the end of the questionnaire; the LSA values obtained in a specific text (which includes both (a) a 
specific justification and (b) the implicit inference ‘i’) could show that this implicit inference is involved in 
the comprehension of the participants. 

It is hypothesized that there is an implicit inference on the basis that the literature (Gigerenzer and 
Hoffrage, 1995) show that, in probabilistic reasoning, natural frequencies are easier to understand than 
relative frequencies, and this is in relation to the fact that a number of consecutive trails in probability have to 
be assumed, as it was emphasized at the beginning of the paper. The implicit inference therefore would be: 
“If I consider, on a consecutive number of trials, a sum of the diverse possible cases in which John shows 7, 1 
see that John shows more times 7 when it is with the Ace than when it is with 8.” 

In order to obtain the LSA cosine values, the informatic tool Coh-Metrix is used; the full long texts with 
justification with/without an implicit inference were introduced. The analyses showed that the difference 
between LSA and LSA_i was marginally significant for the comprehension question Decision, t(95)=2.465; 
p=0.016, and was significant for the question Probability, t(95)=3.648; p<0.01. 

The difference between LSA and LSA_i cosine values seem to show that the implicit inference could be 
more present in the case of the participants who would give the longer explanations N and S. Comparing the 
results of condition C with condition A, there is a significant improvement in correct responses in the 
Probability question; as this increase is only with the increase of number of justifications ‘H’, it cannot be 
said that the overcome of an associative effect is accompanied with the processing of the implicit inference. 
Looking at the large difference in values for the justification H between LSA vs LSA_i, LSA would predict 
that the implicit inference is not necessarily involved in the correct comprehension of the problem. 

LSA seem to able to evaluate more creative explanations as well as both short and large explanations. 
LSA seem to be a helpful tool to evaluate understandable explanations for education. 
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ABSTRACT 

The paper presents a propose of a methodology to treat brain signals in BCI systems. It is an initial work and it tries to 
apply artificial intelligence to solve some problems in the area. 
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1. INTRODUCTION 

The interfaces to human use machines, as keyboard and mouse on computers, cameras and joystics on video 
games, are commonly used by the most of people. However, there is a part of population that they may need 
other ways to interact with electronic equipment due to health problems or accidents, that lead to severe 
paralysis. The development of Brain Computer Interfaces (BCI) systems aims to provide this communication 
without the use of muscle movements. 

BCI systems are enable to communicate based to on neural activity generated by the brain, independent of 
the muscle movements. In the case of people with severe paralysis, but the brain is not compromised, the 
neural activity to perform a movement is generated, but it does not reach the corresponding muscles, or those 
muscles are weak to obey such commands [VALLABHANENI et al., 2005]. 

The potential of BCI systems to help people with disabilities is obvious. There are several designed 
interfaces for people with disabilities [Wickelgren, 2003 apud VALLABHANENI et al., 2005]. Most of these 
systems, however, utilizes some type of muscle control, as the movement of the head, neck, eye, or other 
facial muscles. 

The BCI systems use signals generated voluntarily by the user. Interfaces based on involuntary brain 
activities, as generated during a seizure, utilize many of the same principles and components of BCI systems, 
but are not included in this study area. 

A BCI system has four modules. The first is the acquisition of brain signals, the second is realized pre- 
processing of these signals and the third consists of extracting relevant characteristic signals, and the latter is 
to classify the sets of characteristics. 

This paper present the steps of development of the study and the analysis of brain signals to BCI systems 
using Artificial Intelligence (Al) techniques, especially in the stages of feature extraction and classification. 
In addition to the steps above define, a fifth step is performed, the validation of the classification. 

The paper is strutured in 4 sections. The section 2 presents the main concepts of BCI systems and system 
components, techniques for capturing brain signals and concepts and techniques of feature extraction and 
translation or classification of brain signals. The section 3 presents the proposed methodology to develop BCI 
systems. In section 4, the conclusion are presented. 


2. BCI SYSTEMS 

In 1929, Hans Berger present a device that later became the electroencephalogram (EEG), which could 
record electrical potentials generated by brain activity. After that, some researches speculate that these 
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devices could be controlled using these brain signals. For a long time, it was just a speculation. Just in the 
70's, the first ideas and technologies were developed. 

Nowadays, BCI systems are tools that can help users to communicate and perform daily activities, despite 
a limited success and they are still mainly in research environments. The main users are individuals with 
small to severe muscle deficiency. BCI systems have also been developed for users with certain mental 
disabilities such as autism or for people who do not have any disabilities, primarily as entertainment in the 
gaming industry. 

The goal of a BCI system is to allow the user to interact with a device. This interaction is enabled through 
a variety of intermediary components, control signals and feedback loops, as presented in Figure 1. 
Intermediary components play specific roles to convert the intent into an action. This means that the user and 
the device are also integral parts of a BCI system. The interaction is also possible through feedback loops that 
serve to inform the status of each system component. 
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Figure 1 . Intermediary components and feedback loops in a BCI System [VALLABHANENI et al., 2005]. 

A BCI system has the following modules: 

a) Acquisition of brain signals techniques: 

The brain signals can be captured in invasive or non-invasive way. BCI devices are called “pure” that 
they use cortical implants (electrodes implant into the cranium of the patient). It is an invasive technique. 

There are many non-invasive methods to measure brain activity. Non-invasive techniques reduce the risk 
to users, since it does not require surgery or permanent fixture of electrodes. Techniques such as 
Computerized Tomography (CT), Positron Electron Tomography (PET), Single-Photon Emission Computed 
Tomography (SPECT), Magnetic Resonance Imaging (MRI), functional Magnetic Resonance Imaging 
(fMRI), magnetoencephalography (MEG) and electroencephalography (EEG) have been used to measure 
brain activity non-invasively. 

However, the electroencephalography is the most widely used method for the BCI signal acquisition. 
EEG has a high temporal resolution. It is able to measure the activity every millisecond. Modern devices of 
EEG also have a spatial resolution signals with reasonable, superior to 256 electrodes, simultaneously. 

EEG is used massivaly in laboratories and real-world environments. It is portable and the electrodes can 
be easily placed on the scalp of the user, simply wearing a cap. Furthermore, the EEG systems have been 
used in numerous fields since their creation. 

b) Pre-process of brain signals and extraction of relevant characteristic: 

The study of BCI systems is focused on improving methods to extract features of the collected signals 
and translate them into logical control commands to devices that attend more accurately the user's intent. A 
existent feature in a signal can be seen as reflecting a specific aspect of physiology and anatomy of the 
nervous system [Wolpaw et al., 2000b apud VALLABHANENI et al., 2005]. The goal of feature extraction 
methods is to get the specific physiological aspect of the nervous system through a specific serie. The steps 
involved in feature extraction and translation. 
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The goal of processing and extraction of features (or attributes) is to describe an item through its 
attributes, which should be very similar for the same category, but very different for different categories of 
items. The best solution is when the most relevant characteristics within the numerous options available are 
chosen. 

c) Classification of the sets of characteristics: 

The purpose of training (and classification) is to get that users voluntarily produce EEG signals that are 
detectable and can be altered to achieve a specific result. While the user is not aware to how and when the 
signals are generated, the signal generation process can only be activated by voluntary user actions. However, 
these signals can be produced voluntarily through conscious mental activity, such as adding numbers, or an 
automatic response to a situation, that requires little conscious effort, as riding a bike. 

Translation techniques are developed algorithms to convert the input characteristics (independent 
variable) in commands to control device (dependent variable) [WOLPAW et al., 2002]. Some translation 
techniques which are widely used in other signal processing areas are adapted to BCI. 

There are several types of translation characteristic algorithms. Some use simple features, such as 
amplitude or frequency. Others use individual characteristics, while advanced algorithms use a combination 
of spatial and temporal characteristics. Current algorithms use linear classifiers, common spatial subspace 
decomposition, artificial neural networks, support vector machines and hidden markov models 
[VALLABHANENI et al., 2005], 

Research are being developed by various groups around the world. The work of Kroupi et al. (2011), at 
Ecole Polytechnique Federate Lausanne (EPFL), Switzerland, discloses a method for removing noise from 
eye movements in EEG signals. The study is a comparison of the use of independent component analysis and 
adaptive filtering, concluding that the first is more robust but slower, and it can be used in approaches offline, 
while the second is faster but less accurate and it can be used in some approaches whenever speed is more 
important than the accuracy [KROUPI et al., 2011]. 

Waldert and colleagues (2008), at Universitt Tubingen, Germany, presented a study of EEG and MEG 
signals during movement of the hand in order to analyze the direction of movement. Still using the Shannon 
entropy to quantify the extracted information of the signals. The study showed a modulation in the signals in 
both periods of rest and motion. This modulation was stronger in motor areas of the brain. There were 67\% 
accuracy and concluded that both the EEG signals and MEG can generate good results [WALDERT et al., 
2008], 


3. METHODOLOGY 

A methodology to develop BCI systems is proposed in this paper. It is divided in five steps. In the first step, 
all brain signals are selected. In our tests, we will be used the signals of the PhysioBank (http://physionet.org) 
group. It is a database of biomedical signals and the set of brain signals has 109 healthy peoples (64 electrods 
in each individual), used the 10-20 system, the international distribution of the electrodes on the scalp. These 
signals correspond to the resting state of the person or to do one of the following tasks: i) imagination of 
hand movement; ii) imagination of foot movements; iii) movement of the hands; iv) movement of feet. 

The second step is the pre-processing of brain signals. This step is performed to remove noise, making the 
relevant information present in the signal easier to be identified. EEG signals often have noise from muscle 
movements, especially of the face, as blinking. These movements generate noise that stand out for their 
amplitudes in the EEG signals. The pre-processing of these signals can be done by the application of spatial 
or temporal filters, or more sophisticated filters, as Independent Component Analysis. Besides the removal of 
noise, even in the pre-processing step, the signals can be divided into frequency bands, ie, beta or mu 
rhythms and they can be filtered to be analyzed separately. Both noise removal and separation of the 
frequency bands will be performed using the software BCI2000 (www.bci2000.org). 

The extraction of features of brain signals preprocessed is the third step of the methodology. The amount 
of acquired information in a few minutes is huge, and it increases with the number of used electrodes. In this 
way, it becomes necessary to extract the relevant characteristics of this signal for classification. It is very 
important that the extracted features must be relevant and represent the brain activity required to control the 
desired application, to obtain success in the classification step. The step of feature extraction of signals can 
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be removed from the process and the “pure” brain signals can be passed directly to the classifier, but when 
features were chosen, there is an increase of the performance of classification algorithms. 

Several techniques to BCI feature extraction have been proposed in the literature, which can be divided 
into three groups [GODOI, 2010]: i) Methods that analysis the temporal information in the signal; ii) 
Methods that analysis the information related to the frequency; iii) Hybrid methods, based on time and 
frequency representations. 

The classification is the fourth step and it is the conversion of the characteristics into commands for the 
applications to be controlled. This step may be achieved by regression or classification algorithms. The goal 
of classification is to automatically associate a feature vector to a particular class. This class will identify the 
type of cognitive task performed by the user. The classification algorithms, called classifiers, learn to 
associate the class to the vector feature vector. For example, a data set is recorded from an individual with a 
collection device of brain signals, where that individual must imagine that he/she is moving the right hand by 
a certain time, at another time this individual will imagine that he/she is moving the left hand also for some 
time. These signals, together imagination signals of other movements, will train the classifiers. 

The fifth step consists in to validate the results. To validate the classification capacity of the used 
algorithms, the set of available data is separated into two parts. A party will be used to train the classifier and 
the other will be used to test it. The validation is usually made together with the classification step, where the 
data are separated automatically. A online validation can be done and the brain signals of a person must be 
sent in real time to the already trained classifier, which analyze the data received and return the action desired 
by the user. 


4. CONCLUSION 

Further works in BCI technology should focus on basic and applied research. New resources are needed for 
the user to have more precise and faster control. Algorithms that use artificial intelligence techniques must be 
developed to feature extraction and classification step to become these steps more efficient. Advances in 
methods of signal acquisition, such as invasive techniques are also great promise to increase the spatial and 
temporal resolution. Better and faster training methods will become BCI devices easier to use (popularization 
of the techniques). Increase the consciousness of the potential of BCI technology will be essential to generate 
public interest and government support. 

This paper presented a initial work in the BCI area. The study of BCI systems requires knowledge from 
many different areas such as medicine, computing, electronics, physiotherapy, psychology, and so on, which 
makes it highly interdisciplinary. In this way, to start this kind of work, it is not a easy job. 

The main contribution that we hope of this works is in the extraction and classification steps of 
methodology, focusing in artificial intelligence techniques to solve the open problems in these steps. 


REFERENCES 

GODOI, A. C. B. Detecgao de Potenciais Evocados P300 para Ativagao de uma Interface Cerebro-Maquina. Master in 
Engineer. Escola Politecnica da Universidade de Sao Paulo. 2010. 

KROUPI, E.; YAZDANI, A.; VESIN, J.-M.; EBRAHIMI, T. Ocular Artifact Removal from EEG: A Comparison of 
Subspace Projection and Adaptative Filtering Methods. In: EUROPEAN SIGNAL PROCESSING CONFERENCE 
(EUSIPCO 2011), 2011. 

VALLABHANENI, A.; WANG, T.; HE, B. Brain-Computer Interface. In: HE. B. (Ed.). Neural Engineering. New York: 
Kluwer Academic, 2005. p.85-121. 

WALDERT, S.; PREISSL. H.; DEMANDT, E.; BRAUN, C.; BIRBAUMER, N.; AERTSEN, A.; MEHRING, C. Hand 
movement direction decoded from MEG and EEG. Journal of Neuroscience, [S.L], v.28, n.4, p.1000-1008, 2008. 

WOLPAW. J. R.; BIRBAUMER. N.; MCFARLAND, D. J.; PFURTS CHELLER. G.; VAUGHAN. T. M. Brain-computer 
interfaces for communication and control. Clinical neurophysiology, [S.L], v.113. n.6, p.767 - 791, june 2002. 


256 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


A PROJECT-BASED LEARNING SETTING TO 
HUMAN-COMPUTER INTERACTION FOR TEENAGERS 


Cornelia Geyer and Stefan Geisler 

Hochschule Ruhr West, University of Applied Sciences, Tannenstr. 43, Bottrop, Germany 


ABSTRACT 

Knowledge of fundamentals of human-computer interaction resp. usability engineering is getting more and more 
important in technical domains. However this interdisciplinary field of work and corresponding degree programs are not 
broadly known. Therefore at the Hochschule Ruhr West. University of Applied Sciences, a program was developed to 
give teen-aged pupils insights into this area in a project-based learning environment with professional tools. Within the 
last 18 month this project was successfully conducted several times with participants of different age. 
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1. INTRODUCTION 

In Germany, the number of beginners of technical studies and especially computer science is comparably 
low, though the employment situation in this area is quite good. For this reasons several initiatives have been 
started, to demonstrate the wide range of activities in technical studies and jobs. Several universities 
cooperate with schools and organize open days, guest lectures of professors in schools or project days in 
universities. 

One additional aspect is the advancement of women in technical jobs. In Germany, every year in April the 
Girl's Day is declared nation-wide, where the school girls can visit companies or universities and especially 
get an introduction to occupational fields which are normally dominated by men. 

Within this situation we wanted to define a project which gives insights into an area of computer science 
which is presumably attractive to young pupils who are interested in technology but also human sciences 
and/or design to show that computer science is more than pure coding. We chose a project-based approach 
(Blumenfeld, 1991) as we wanted to offer an experience which differs from normal school days, results in a 
product, that can be taken home and that stays in mind for a longer time though the project execution time is 
rather short. 

In the same project, professional tools and workflows should be presented to the participants as these 
normally are not available in schools and to demonstrate a more realistic working environment than academic 
or especially educational software development tools (see Terwelp and Dahm, 2011 for an overview) could 
do. On the other hand, we wanted the group to work on their own and create software which would run on 
their own computer or smartphone and have good memories on this day. Last but not least the project should 
be able to be carried out with participants of different ages and different levels of previous knowledge in 
computer science or programming. 


2. FIELD REPORT: UI-DEVELOPMENT WITH TEENAGERS 

The project “My personal MP3-Player” was executed several times since April 2011 at the Hochschule Ruhr 
West, University of Applied Sciences with 44 pupils aged 14-19. These workshops were integrated in 
different forms of cooperation with schools or nationwide programs like Girl's Day. 
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2.1 The MP3-Player Software 

To allow participants with different knowledge in computer science to work in this project, a tool was 
required that allows the UI development without any programming, but also should have the possibility to 
discuss and change source code if wanted. The Adobe Flash Builder 4.5 was chosen, as it fulfills this 
requirement and is due to the Flash base more flexible on the graphical side than other GUI development 
tools. Furthermore this tool allows platform independent development inch Windows-PC, Mac OS X, 
iPhone/iPad and Android devices. 

With this tool a MP3 -Player software was developed in advance in a way, that the UI can easily be 
modified, so that everyone can create the desired look & feel without the need of programming. 
Programming only is required if the behavior should be changed. The software structure therefore is prepared 
to support this with quite low effort. 

To avoid that participants stay close to the start configuration of the UI, the buttons and labels are 
completely disordered, every button has a different style, every text a different color, etc. Figure 1 shows this 
configuration. Though the interface looks very confusing it is fully functional. 


2.2 The Workshop 


At the beginning of the workshop a short introduction into the field of human-computer interaction was 
given. Jointly the typical controls and displayed information of MP3 players were discussed. The topic of 
meaningful grouping of elements was dealt with by using cards showing the controls and displayed 
information. The group was asked to organize these cards in a reasonable order at the blackboard. In addition 
to this UI introduction some options of image editing were shown. 

The first phase of creative work in this project was performed unstructured by intention. The participants 
worked depending on their preference alone or in pairs on the user interface of a MP3-Player software for 
desktop PCs or Android phones. By this, every participant was able to develop own ideas in own speed. To 
support the realization with the software tools two tutors for eight groups were present. Button graphics were 
downloaded in most cases from Web Sites or self-drawn. Downloaded graphics were edited (mainly change 
of colors and cropping single buttons out of images) with either Gimp or Photoshop. 

The given user interface was modified by using the graphical editor of Adobe Flash Builder 4.5. Changes 
to the prepared behavior could be realized on a limited scale with support of the tutors. But this kind of 
requests occurred only rarely, presumably due to the fact, that the behavior of an MP3 player is in main 
aspects standardized. Depending on the interest, age and knowledge of the pupils parts of the source code 
were discussed with the participants. 

Figure 1 shows the start situation for the participants and pictures from the workshops. 



Figure 1 . Pictures from the workshop and start configuration of the MP3 player 


2.3 Results of the UI Development 

The pupils developed the user interface of an MP3-Player according to their own preferences within 
approximately four hours. The main focus was on the appearance and styling, but also different questions of 
interaction concepts were considered. 
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The results have a high variety in their appearance, but are quite similar in functional behavior. This is 
likely due to the case, that a well-known stereotype for operating music players exists, which was discussed 
in the introduction to the project work. 

Another observation is related to the gender. Girls were more focused on aesthetics and the design of the 
overall concept while boys often tried to bring in symbols or graphics related to their interests or hobbies 
(e.g. favorite soccer club). Figure 2 shows different outcomes of the project. 




Figure 2. Different outcomes from the project day 


3. EVALUATION 


After some of the executed workshops a survey was carried out. The results of the N=24 pupils who took part 
in the evaluation overall was very positive. Especially they liked the possibility of creativity in the project 
and the practical work at the computer. This strengthens our approach of offering project-based workshops 
instead of pure trial lectures to elate someone. 

The overall average results on a five-level Likert scale with 1: very good, 5: insufficient of the most 
important items were: 

• I could gather a first insight into the topic: 1:42%, 2:58% 

• The project course fulfilled my expectations: 1:33%, 2:50%, 3:17% 

• The content of the project course was interesting: 1:63%, 2:38% 

• The content was comprehensibly communicated: 1:58%, 2:42% 


Content of the project course was interesting 


Content and level of project course was 


The course arouse my intererst in c.s. / hci 


0,9 

0,R 

0,7 

0,fi 
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Figure 3. Feedback comparison of pupils with and without courses in computer science (c.s.) at school. 


Looking more into detail the comparison of pupils with courses in computer science at school (N=13) 
with those who don’t (N= 11) for most interesting items are shown in figure 3. 

The discussion of these results is within the limitation of the small sample size, but is enriched by verbal 
comments given back over the teacher a few days after the workshop. So they give a first indication how to 
improve the course. 

In average it shows that the flexible concept of the workshop, which allows adaptable amount of work 
with the source code, is suitable for participants with and without school knowledge in computer science. 
Some of the participants with already higher knowledge in computer science tend to prefer some deeper 
programming work than was offered so far. On the other hand this needs to be done carefully so that people 
with less knowledge will not get lost. 
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Another interesting item to evaluate is, if the course content is able to arouse interest in computer science 
and in human-computer interaction in particular. As the third graph in figure 3 shows this is the case. 
Especially the person with less knowledge (and lower age) learned more about different aspects of the tasks 
of software engineers. Positively mentioned by a high number of participants was the practical work, self- 
reliant in defining the goal and technically as long as possible, but getting support whenever required and at 
the end having their own software to take home. 


4. IMPLICATIONS FOR USABILITY ENGINEERING PROCESSES 

The success of working together with teenagers in a professional software tool for user interface 
programming could also have some implications on the usability engineering process (Mayhew, 1999) for 
software targeted for this group. 

A participatory design process (Muller, 2003) seems applicable in a manner that professional UI design 
software can be used with the teenagers. Normally either low-fidelity prototyping, e.g. showing screens of 
the user interface on paper, is used or a special software version is created. The first variant has the 
disadvantage that the important user experience (Buchenau and Fulton Suri, 2000) differs significantly from 
the final product, the latter one that additional effort in programming is required. 

Further research will be done to explore the range of products and the depth of interaction design which 
can be covered with teenagers in such a project. 


5. CONCLUSION 

The developed project “My personal MP3-Player” was performed successfully several times to inform and 
elate teenagers about topics in the area of computer science and especially human-computer interaction. The 
project-based approach with professional design and software development tools was according to the 
participants’ feedback very well accepted. The goal to arouse interest also was achieved. 

In further modifications of the project the aspect of integrating programming blocks for participants with 
previous knowledge need to be worked out. The current project duration is around four hours. The base 
concept of the workshop allows extending the duration by including other parts of the usability engineering 
life cycle like user testing into the project. Both together give the opportunity for a long-term project or 
regular course over a complete semester to hold contact between pupils and university. Details need to be 
worked out and evaluated. 

With the success of this project, another research topic for the future will be the integration of teenagers 
into the usability engineering process. 
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ABSTRACT 

An underlying theme of this paper is that it can be easier and more efficient to conduct valid and effective research 
studies in online environments than in traditional classrooms. Taking advantage of the “big data” available in an online 
university, we conducted a study in which a massive online database was used to predict student successes and failures. 
We found that a pattern of declining performance over time is a good predictor of the likelihood of dropping out, and that 
having dependents or being married or in the military reduces the risk of dropping out. The risk of dropping out was 
higher for older students, females, and students with previous college education or transfer credits. These results provide 
a foundation for testing interventions to help students who are at risk and will also help to inform the development of a 
"research pipeline” that will enable rapid experimental studies of new tools and strategies. 


KEYWORDS 
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1. INTRODUCTION 

The early years of the 21 st century have witnessed an explosion of interest in new technologies for learning 
and teaching at all levels of education. Notable examples of applications of technology in education include 
the Khan Academy and the massive open online courses (MOOCS) offered by edX, Coursera, Udacity and 
other entrepreneurial partnerships. Non-profit and for-profit online universities now provide a wide variety of 
postsecondary ttaining, certificates, course credits, and degrees, and traditional colleges and universities have 
greatly expanded their online learning opportunities. These and a startling proliferation of other online 
opportunities have made it possible for students and educators to pursue a range of alternative learning and 
teaching paths not available to them before. 

Whether new forms of online instruction can be effective in educating students on a large scale across the 
world remains a vexing question, however. At the moment, the evidence is lacking, but the opportunity to 
build evidence does exist. The growth of online instruction across the globe in fact opens new opportunities 
for data collection, analysis and reporting and for studies of the effectiveness of research-based instructional 
strategies, among other things (e. g.. Baker, in press; Campbell et al., 2007; Cen et ah, 2006; Desmarais et ah, 
1996; Means et ah, 2010; Romero et ah, 2008; Romero et ah, 201 1). 

These opportunities include the possibility of using routinely collected data to evaluate the relative 
effectiveness of current and innovative instructional approaches, as well as the opportunity to provide better 
information to guide learners and their teachers. In online environments it is possible to collect detailed 
real-time data on every action taken by every learner. The existence of these data for thousands, tens of 
thousands, and even millions of students studying the same topics or using the same curriculum under 
different conditions, gives us new leverages for studying the influence of contextual factors on learning and 
learners. This “big data” affordance can help learners by identifying which learning paths might be best for 
them, teachers by recommending approaches for helping students who are struggling, and researchers by 
enabling them to test principles of learning and instruction in authentic learning environments at scale. 

As a first step toward capitalizing on these opportunities, we conducted an initial investigation intended to 
use large existing datasets to predict student success and failure in an online university program. Having 
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developed algorithms to flag students needing additional support, we then plan to use both data mining and 
experimental methods to test which types of support will be most effective and efficient for which students. 


2. BODY OF PAPER 

2.1 Analyzing “Big Data” to predict Student Dropouts 

The study we conducted involves the mining and analysis of “big data”, which in our case refers to large 
existing datasets that can be analyzed to discern patterns in student and teacher performance, identify at-risk 
students, and study relationships among important variables, such as attendance, learning, and student 
satisfaction. 

We conducted this study in an online university that offers approximately 1000 different online courses to 
about 60,000 students and that has assembled an extensive database of information on students and their 
performance. This university tends to enroll students who are older and for various reasons need an 
alternative to a traditional classroom-based university program. 

The database we analyzed contains student and faculty background data as well as measures of learning, 
student satisfaction, retention, engagement, teacher performance, and postgraduate success. These data make 
it possible to test a wide range of research and evaluation questions on a very large scale, as well as to 
monitor and respond to student performance, engagement, and motivation. 

The study was specifically designed to examine how students’ academic and demographic characteristics 
relate to their dropout rates. We analyzed academic and demographic characteristics of degree-seeking 
students (N = 14791) enrolled during a two-year period in the online university. Demographic variables 
analyzed were age, gender, marital status, military status, previous college education, estimated family 
financial contribution, and number of transfer credits from other universities. Academic variables included 
measures of student performance available in the online eCollege platform, such as final exam, discussion, 
project, and other assignment scores. 

Survival analyses (a type of logistic regression analysis) revealed that measures of student performance 
that declined over time were significant predictors of the likelihood of dropping out. With respect to the 
demographic predictors, we found that age was a significant predictor of retention, with older students more 
likely to drop out. Military and married students retained at a higher rate than non-military and unmarried 
students, respectively, and students with prior college experience and higher financial contributions from 
their families retained at lower rates than students without these characteristics. 

Results of the analyses are presented in Table 1, with the last column showing the effect of each predictor 
variable on the risk of students dropping out. The outcome we modeled is student retention, coded as a 
dichotomous variable with students who dropped out receiving a “1” score on the variable. 

For comparison purposes we also calculated odds ratios based on a logistic regression analysis; this 
analysis produced the same outcomes as the survival analysis. 


262 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


Table 1. Predicting student dropouts 




Parameter 

Standard 

Chi- 

Pr > 

Hazard 

Effect on 

Predictor 

DF 

Estimate 

Error 

Square 

ChiSq 

Ratio 

dropout risk 

Age between 29 and 38 

1 

-0.50719 

0.04485 

127.9116 

<.0001 

0.602 

Reduces risk 
by 40% 

Age between 38 and 45 

1 

-0.07808 

0.04767 

2.6833 

0.1014 

0.925 

Reduces risk 
by 7.5% 

Older than 45 

1 

0.26797 

0.04479 

35.7943 

<.0001 

1.307 

Increases risk 
by 30.7 % 

Has transfer credits 

1 

0.86241 

0.04427 

379.5799 

<.0001 

2.369 

Increases risk 
by 236 % 
Increases risk 
by 7.4% 

Enrolled in 200 level courses 

1 

0.07112 

0.07777 

0.8365 

0.3604 

1.074 

Enrolled in 300 level courses 

1 

-0.31259 

0.07711 

16.4345 

<.0001 

0.732 

Reduces risk 
by 26.8% 

Enrolled in 400 level courses 

1 

-1.07894 

0.08296 

169.1268 

<.0001 

0.34 

Reduces risk 
by 66% 
Reduces risk 
by 72.8% 

In the military 

1 

-1.3035 

0.08964 

211.4631 

<.0001 

0.272 

Previous college education 

1 

0.12529 

0.04324 

8.3962 

0.0038 

1.133 

Increases risk 
by 13.3 % 

Female 

1 

1.20342 

0.06681 

324.4385 

<.0001 

3.331 

Increases risk 
by 330 % 
Increases risk 
by 17 % 

Estimated financial contribution 
from family 

1 

0.15667 

0.03569 

19.275 

<.0001 

1.17 

Married 

1 

-0.44225 

0.03807 

134.98 

<.0001 

0.643 

Reduces risk 
by 35.7% 

Has dependents 

1 

-0.86645 

0.05448 

252.9138 

<.0001 

0.42 

Reduces risk 
by 58% 

Above median score on 
discussions 

1 

-0.72133 

0.0497 

210.6254 

<.0001 

0.486 

Reduces risk 
by 51.4% 

Above median score on final 
exam 

1 

-1.4674 

0.16955 

74.9057 

<.0001 

0.231 

Reduces risk 
by 76.9% 

Above median score on projects 

1 

-0.96057 

0.06153 

243.6801 

<.0001 

0.383 

Reduces risk 
by 61.7% 

Above median score on course 
review 

1 

-1.04862 

0.05681 

340.7227 

<.0001 

0.35 

Reduces risk 
by 65% 

Above median score on other 
teacher-graded assignments 

1 

-0.98675 

0.05474 

324.9456 

<.0001 

0.373 

Reduces risk 
by 62.7% 


3. CONCLUSION 

3.1 Moving Toward Rapid Experimentation 

The finding that declining performance over time is related to dropout tendencies, while not surprising, is a 
particularly useful one for us. It represents a first step on the path toward generating data for faculty and 
administrators that will enable them to provide additional support to students who have a high likelihood of 
failing or dropping out. Results on student background variables will serve a similar function, enabling us to 
build profiles of students who may need more support to succeed in the program. Some of our findings for 
these variables were surprising; for example that females, unmarried students and students with transfer 
credits or dependents are more likely to drop out. We can speculate about these relationships but it will be 
more useful to conduct follow-up investigations, including qualitative studies with representative samples of 
students, to explore why some groups drop out at higher rates. 

As we continue to collect data over time we can also confirm or disconfirm the strength of the 
relationships we have found and possibly discover new ones. We are currently developing prior knowledge 
measures in several courses and expect, based on extensive previous research on the effects of prior 
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knowledge on learning (e. g., National Research Council, 1999; Sweller et al, 2011), that these will be strong 
predictors of student success, failure and retention in online courses. Ultimately we will be able to determine 
which combinations of indicators and performance patterns constitute “red flags” requiring immediate, strong 
intervention and which may call for less intensive strategies. 

As valuable as these analyses of routinely-collected big datasets may be, however, they give us only part 
of the information needed to build more effective evidence-based educational programs. Another critical 
piece is the testing of new instructional, motivational and support strategies to help students who are having 
difficulty and may be likely to drop out. To determine what kinds interventions will be most effective for 
which students will require experimental studies in which students are randomly assigned to different 
instructional conditions, so that alternative hypotheses for observed changes or differences in student 
performance (such as differences in initial student ability levels) can be ruled out. In this case we can take 
advantage of another technology affordance for research: online learning systems make it possible to run 
large numbers of randomized control trials far more rapidly than would otherwise be possible (Shadish and 
Cook, 2009). 

We are currently developing “research pipelines”, or platforms and processes that will to enable us to 
randomly assign individual students, class sections, or classes to different instructional experiences. The 
pipelines will make it possible to build up knowledge on scalable improvements incrementally, testing the 
impact of one variable at a time, but quickly over time. Big data analytics in combination with experimental 
testing will thus enable us take advantage of the tremendous scale of the online university to find out what 
works more quickly than would otherwise be possible-and have the data to prove it. (As a final note, it is 
worth mentioning, however, that the online university in this study primarily serves a high percentage of 
students who for one reason are another are not able to attend or are not interested in attending traditional 
college and university programs, so results may not generalize to students in those programs.) 


REFERENCES 

Baker, R.S.J.d, in press. Data Mining for Education. To appear in McGaw, B., Peterson, P.,Baker, E. (Eds.) International 
Encyclopedia of Education (3rd edition). Elsevier, Oxford, UK. 

Campbell, J. P., DeBois, P. B., and Oblinger, D. G., 2007. Academic Analytics: A New Tool for a New Era. EDUCAUSE 
Review. EDUCAUSE, http://net.educause.edu/ir/library/pdf/erm0742.pdf 

Cen, H., Koedinger, K., Junker, B., 2006. Learning Factors Analysis - A General Method for Cognitive Model Evaluation 
and Improvement. Proceedings of the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, 
pp. 12-19. 

Desmarais, M.C., Maluf, A., and Liu, J., 1996, User-expertise Modeling with Empirically Derived Probabilistic 
Implication Networks. In User Modeling and User-Adapted Interaction, Vol. 5, No. 3-4, pp. 283-315. 

Mayer, R. E., amd Alexander, P. A. (Eds.), 2011. Handbook of Research on Learning and Instruction. Routledge, New 
York, USA. 

Means, B., Toyama, Y., Murphy, R„ Bakia, M. & Jones. K., 2010. Evaluation of Evidence-based Practice in Online 
Learning: A Meta-analysis and Review of Online Learning Studies. US Department of Education Office of Planning, 
Evaluation, and Policy Development, http://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf 

National Research Council, 1999. How People Learn. National Academy Press, Washington DC, USA. 

Romero, C., Ventura, S., Pechenizkiy and Baker, R.S.J.d, (Eds.), 2011. Handbook of Educational Data Mining. CRC 
Press, Boca Raton, FL, USA. 

Romero, C., Ventura, S., Espejo, P.G., Hervas, C., 2008. Data Mining Algorithms to Classify Students. Proceedings of 
the First International Conference on Educational Data Mining. Montreal, Canada, pp. 8-17. 

Shadish, W. R. and Cook, T. D., 2009. The Renaissance of Field Experimentation in Evaluating Interventions. In Annual 
Review of Psychology, Vol. 60, No. 1, PP- 607-629. 

Sweller, J., Ayers, P., & Kalyuga, S., 2011. Cognitive Load Theoiy. Springer, New York, USA. 


264 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


EFFECTS OF PEER-TUTOR COMPETENCES ON 
LEARNER COGNITIVE LOAD AND LEARNING 
PERFORMANCE DURING KNOWLEDGE SHARING 


Ya-Ping (Amy) Hsiao, Francis Brouns, Jan van Bruggen and Peter B. Sloep 

Centre for Learning Sciences and Technologies (CELSTEC), Open University of the Netherlands 
P.O. Box 2960 6401 DL Heerlen The Netherlands 


ABSTRACT 

In Learning Networks, learners need to share knowledge with others to build knowledge. In particular, when working on 
complex tasks, they often need to acquire extra cognitive resources from others to process a high task load. However, 
without support high task load and organizing knowledge sharing themselves might easily overload learners’ limited 
cognitive capacities because learners first have to find relevant peer tutors (i.e., those who provide help) and then 
maintain the social interaction. We propose to design a peer-support system that selects tutors and provides support 
during knowledge sharing. The pilot study reported here investigated the effects of two peer tutor competences, tutoring 
skills vs. content knowledge, on tutees’ (i.e., those who need help) cognitive load and learning performance. The results 
show that tutees supported by tutors with tutoring skills experienced lower cognitive load and had better learning 
performance than did tutees supported by tutors with content knowledge. This is in line with our assumption, but for 
confirmation we need to gather more data in a full study. We need to first use a task that requires learners to rely on 
others to trigger higher cognitive skills to deal with high task load. Secondly we need to find a modus to ensure that the 
tutors follow the instructions to apply the particular competence. 


KEYWORDS 

Peer-tutor competence, content knowledge, tutoring skills, knowledge sharing, cognitive load, tutor selection. 


1. INTRODUCTION 

A Learning Network (LN) is a particular kind of online social network that is dedicated to learning (Sloep 
2009). In LNs, learners are themselves responsible for sharing knowledge with their peer learners. When 
organizing knowledge sharing themselves, learners first have to find out relevant knowledge sharers and then 
maintain the social interaction with others to reach a shared understanding and knowledge building. To help 
finding relevant knowledge sharers, our colleagues have developed two tutor selection systems that 
automatically assign peer learners with relevant content knowledge to act as peer tutors to help answer each 
others’ content-related questions (Van Rosmalen 2008; De Bakker 2010). However, due to the heterogeneous 
group composition of LNs it is not always possible to find tutors with content knowledge related to tutees’ 
questions. In addition, the questions or problems that participants of LNs have are often rather complex: they 
are authentic problems originating from real-life contexts, such as the working place. When working on 
complex tasks, learners have to allocate many of their cognitive resources to process numerous information 
elements and element interactivity and this imposes a high cognitive load (Sweller et al. 1998). Cognitive 
load (or mental workload) refers to the learner’s limited cognitive capacity actually allocated on performing a 
particular task and it has been recognized as an important factor that influences learner performance (Sweller 
et al. 1998; Hart 2006). We surmise that tutors who provide content-related knowledge only cannot alleviate 
their tutees’ cognitive load. 

Furthermore, maintaining the social interaction to reach a shared understanding or build knowledge on 
complex tasks requires certain pedagogical and process -facilitation skills (King 2007; Roscoe and Chi 2008). 
These skills go beyond the knowledge transfer or information exchange that content tutors often offer. If it is 
not possible to always find tutors with content knowledge precisely related to tutees’ questions, nor do tutors 
with content knowledge guarantee effective tutee learning, it is therefore necessary to consider tutors with 
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tutoring skills. However, while most tutoring studies have examined effects of tutors’ content knowledge on 
diverse dependent variables of tutees, there have not been many studies that measured effects of tutors’ 
tutoring skills on tutees’ learning. Studies of reciprocal peer tutoring have compared the effects of supporting 
learners with and without tutoring skills on learner performance (King 1994; King et al. 1998; Nath and Ross 
2001). Their support of tutoring skills was to make sure that learners would demonstrate tutoring skills to 
elicit certain social interactions and trigger cognitive processes that contribute to learning. Some studies have 
shown that the learners with tutoring skills outperformed the control groups on knowledge or comprehension 
tests (King 1994; King et al. 1998). However, in LNs knowledge sharing involving peer support is more akin 
to asymmetric rather than reciprocal peer tutoring. From these studies we can therefore only speculatively 
claim that tutoring skills are likely to promote tutee learning. Considering human limited cognitive capacities, 
we need to redesign our previous tutor selection system to figure out what makes an effective tutor and to 
design support for the social interaction process. To achieve this, we first of all need to investigate the effects 
of peer-tutor competences, i.e. content knowledge vs. tutoring skills, on tutee learning. In this pilot study we 
investigate whether it is possible to determine an effect of the two types of peer-tutor competence when 
tutors do not receive prior training but instead receive a tutoring guide to follow. 


2. METHOD 

Two computer science classes jointly consisting of 28 students from a pre-university secondary school in the 
Netherlands took part in this pilot. We assigned students in half of classes to act as tutors and the other half as 
tutees. The tutor role was split; a tutor was either a tutoring skills (TS) or a content knowledge (CK) tutor. In 
total there were seven TS and seven CK pairs. The intervention of this study was to provide tutors with a 
tutoring guide including additional course materials (for CK tutors) or instructions of tutoring skills (for TS 
tutors) to assure that tutors had either CK or TS competence. The instructions that TS tutors received 
consisted of general rules for pedagogical and process-facilitation skills as well as specific step-by-step 
directions that guide tutors to perform pedagogical and process -facilitation skills (King 2007). This study 
consisted of two sessions in 2.5 hours. Two websites were set up for these two sessions: the course site 
contains the learning material of Sex and Evolution as well as the task site contains task instructions, chat and 
wiki tools. During the first session, students studied the learning material on the course site and then they 
took the pre-test and filled in the Tutoring Skills Questionnaire. There was a 15-minute break between these 
two sessions. The second session was started with a brief introduction of peer tutoring, task requirements and 
use of chat and wiki tools. Then students had 50 minutes to work in pairs to complete the essay task. While 
performing the task, students no longer could consult the course material. When working on the task, 
students had to use a chat to communicate with each other and they were not allowed to talk face-to-face. 
Tutees had to write the essay in a wiki and to “publish” the wiki page frequently to allow the tutors to read up 
on the tutee’ s progress. Tutors could only read the wiki page, but they could not edit it. To view the updated 
wiki page, tutors had to “refresh” the wiki page frequently. Tutors used a respective tutoring guide to help 
tutees work on essay answers through the chat. After the task, students had 20 minutes to indicate 
experienced cognitive load on the NASA-Task Load Index (see Table 1) (Hart 2006), to take the post-test 
and to fill in an evaluation survey. 


3. FINDINGS 

Pre-measures. Both groups of tutees estimated their tutoring skills to be quite high, 63 out of possible 75. 
The low pre-test scores showed that tutees of both groups did not have sufficient prior knowledge though TS 
tutees scored slightly higher than CK tutees. For both pre-measures, there was no difference between tutees 
assigned to TS or CK pairs. 

Tutee learning performance. Tutees' low post-test scores indicated that working on the essay task did 
not help students leant much related to the course materials. In addition to the same items of the pre-test, 
there were five task-related items in the post-test. CK tutees performed better on five task-related items than 
TS tutees: this means that additional course materials might help tutees understand the topics included in the 
essay task better than instructions of tutoring skills. Both groups of tutees performed equally well on essay 
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answers though TS tutees performed slightly better than CK tutees. Qualitatively, TS tutees answered more 
topics completely than CK tutees whereas CK-tutees answered more topics partially than TS-tutees. 

Tutee cognitive load. TS tutees’ total cognitive load is lower than CK tutees’ and lower on all six 
dimensions. In particular, TS tutees experienced much lower cognitive load on physical demand and 
frustration than CK tutees. 

Table 1. Tutee cognitive load on six dimensions of NASA-TLX 




Mental 

demand 

Physical 

demand 

Temporal 

demand 

Performance 

Effort 

Frustration 

Total 

TC 

Median 

12.00 

3.00 

10.00 

6.00 

9.00 

2.00 

41.00 

1 N 

SD 

4.27 

3.82 

6.05 

3.70 

4.08 

5.68 

14.60 


Median 

14.00 

11.00 

10.50 

8.00 

13.00 

10.50 

68.00 

v K 

SD 

5.59 

4.32 

4.66 

4.67 

3.93 

2.78 

20.01 


Tutee cognitive load and learning performance when their tutors applied the tutoring guide. Chats 
were analyzed for evidence of adherence to the assigned tutoring competence. Only two of the TS tutors 
applied the specific step-by-step instructions and five of the CK tutors referred to the additional course 
material. Table 2 shows that TS tutees performed better on all performance measures and they experienced 
lower cognitive load than CK tutees. 

Table 2. Performance measures and cognitive load of TS tutees and CK tutees supported by tutors who actually applied 

intervention during chats 


TS-tutees (n = 2) CK-tutees (n = 5) 



M 

SD 

M 

SD 

pre-test 

5.00 

2.83 

3.75 

.96 

post-test 

6.50 

3.54 

4.40 

1.52 

5 task-related items 

4.00 

0 

3.60 

1.67 

essay 

7.92 

1.29 

6.71 

1.78 

total cognitive load 

51.00 

14.14 

68.20 

10.85 


4. CONCLUSION AND DISCUSSION 

Although TS tutees had higher scores on the essay task and post-test, CK tutees seemed to have higher scores 
on the five task-related items than TS tutees. From how multiple topics were answered in the essays, we 
surmise that the different tutor competences influenced how tutees worked on the task. The diverse examples 
included in the additional course materials helped CK tutees answer more topics partially while the 
instructions that TS -tutors used helped TS tutees answer topics more completely. When looking into the data 
of tutees supported by tutors who indeed applied the intervention of additional course materials and tutoring 
skills, the results in Table 2 turned out to be different from the group data: TS tutees performed better than 
CK tutees on both test and task measures. During knowledge sharing, CK tutees in general experienced more 
cognitive load than TS tutees. In particular, the much lower score on frustration corresponds to findings of 
our previous study (Hsiao et al. in press). We observed that some CK pairs completed the task much earlier 
than the expected time of 50 minutes. Although using additional course materials helped CK tutees complete 
the task earlier, CK-tutees still indicated a high cognitive load. 

The findings of this pilot are hopeful in the sense that discerning CK and TS tutoring competences seems 
to make sense. In order to corroborate our first findings, we need to improve three aspects for the full study. 
First, in this study we tried out different performance measures in order to find out which could clearly 
answer the research question in the full study. The pre- and post-test that measured students’ understanding 
of the entire course could not appropriately gauge the effects of the intervention on the essay task since this 
essay only covered several topics of the course. Thus, to better measure learning performance, we should 
either expand the number of task-related items to a longer test or use the essay answers that directly reflect 
how students learn from working on the task. Second, tasks with different complexities require different 
amount of cognitive resources to deal with the task load. Complexities of essay questions not only depend on 
the interactivity of information elements but also on the level of cognitive skills required. The essay question 
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of this study is still a relatively simple task: comparing and contrasting facts require a lower level of cognitive 
skills, namely understanding. From satisfactory essay scores, we surmise that tutees might be able to answer 
the essay by merely retrieving learned factual knowledge from their own or their tutors’ memory instead of 
performing higher cognitive skills to process the task load. This corresponds to our model that when working 
on simple tasks learners do not need to resort to knowledge sharing or that knowledge sharing does not 
contribute to better learning effects (Hsiao et al. 2011). Third, considering the online characteristics of LNs, 
we supported tutors with certain competences by giving them ready-to-use tutoring guides instead of giving 
them a prior training, as is commonly done in online collaborative learning studies or peer tutoring in face-to- 
face classrooms. The results showed that five of seven CK tutors and two of seven TS tutors and applied the 
respective intervention. Our findings confirm the findings of peer tutoring studies: without training, peer 
tutors seldom demonstrate certain skills to fulfill their role tasks (Nath and Ross 2001). Our observations 
during the experiment might explain why TS tutors did not use tutoring skills: i) tutors might not have 
sufficient time to read all of the textual instructions, ii) tutors might not realize the relevance of using the 
general instructions with answering the essay question, and iii) students might have developed certain 
internal scripts of dealing with such essay questions comparing facts and thus they did not need to use our 
intervention. 

To conclude, for the full study we need to oblige tutors to apply the support of competences as we 
expected, in particular for tutors to apply tutoring skills. In addition, we should implement the full study with 
a more complex task that requires higher-order cognitive skills such as analysis, evaluation and synthesis, to 
make sure that learners benefit from the kind of knowledge sharing with others that triggers extra cognitive 
processes. 
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ABSTRACT 

The Greek Ministry of Education has initiated the project “Digital School” since September 2011. One of the actions of 
the Digital School concerns the development of digital interactive books (in html) for all school courses. These 
interactive books are enriched with digital components with activities which are embedded in them. The Course 
Management System ‘eClass’ was selected as the platform to support the project and to provide the place for 
communication and collaboration among the teachers participating in the project. It is supported that teaching with such 
interactive books offers some advantages such as offering to the students more opportunities for engaging and interacting 
with the books content, multiple representations of the content information, which are expected to result to the better 
imprint in the students' memory. We envisage that this kind of interaction with the content of the books of Religion will 
create favorable conditions for the teaching approach of the course of Religion and it will be an initiative for renewal of 
its teaching strategies. 


KEYWORDS 

Interactive book, education, religious education, interactive components 


1. INTRODUCTION 

The Greek Ministry of Education since September 2011, has initiated the project “Digital School'’. 
(http://digitalschool.minedu.gov.gr/) One of the actions of the project concerns the design and development 
of digital interactive books. The interactive books consist an additional form of the e-books already uploaded 
on the e-platform of the greek ministry of education and can contribute to the better exploitation of the 
interactive boards in the classrooms. The interactive books are in html format, and on them are embedded 
various digital components with activities e.g. Interactive maps, crosswords, quizzes, concept maps, 
interactive exercises, power point presentations, links to wikipedia, dictionaries and other similar sources etc. 
The digital components appear as small pictures on the interactive book. When the cursor is moved on the 
picture, a pop-up menu opens with all the information of the component (title or kind of the activity (e.g. 
Concept map). When clicking on it, a window opens with the activity and information about it e.g. Chapter, 
lesson, title, guidelines of use etc). (Kapaniaris & Papadimitriou, 2012) 


2. E-PLATFORM OF THE ‘DIGITAL SCHOOL’ 

As place for the actions of the project “Digital School” was selected the Course Management System 
‘eClass’ (digitalschool.minedu.gov.gr). It is used as platform where there are uploaded the digital 
schoolbooks ( already in pdf and gradually in html format) of all courses and it provides a common space for 
communication and collaboration of the teachers working on the design and development of the digital 
components. The html format of the books will render easier on one hand their enrichment with the digital 
components (interactive maps, digital map etc) and thus their change into interactive. 
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Among the aims of this project are: (a) to provide the greek students the possibility to have access to the 
digital and interactive schoolbooks from their homes, at their free time, so that they will be able to practice 
with the activities at their own pace and time (b) to provide the teachers the possibility to have immediate 
access to a great variety of teaching material so as to be able to use it for teaching with interactive whiteboard 
but also to enrich and upgrade their teaching approach. (Mitropoulou, 201 1) 

‘eClass’ Course Management System is used to facilitate the communication and collaboration among the 
teachers working on the development of the interactive books. There are 10 teacher-teams working on the 
following courses: Modern Greek Language, Physics, English, Informatics, Mathematics, French, Music, 
Chemistry, Geography and Religion. The field of the platform used for this purpose is called” 
“Synergazomai-Symmetexo” (Collaborate-Participate) (http://synergasia.minedu.gov.gr/) and provides to all 
teams a place to communicate, upload and store the components/activities they design and develop. Then the 
components are incorporated as links on the html schoolbook, which, finally, is uploaded on the platform of 
the Digital school. 


3. THE INTERACTIVE E-BOOKS FOR THE COURSE OF RELIGION 

All the schoolbooks for Religion both for Primary and Secondary Education are already uploaded on the 
eClass platform of the Digital School. Up-to-day, those which have been enriched are the books for 
Secondary Education (High School: 1 st , 2 nd , 3 rd grades, and Upper High School [Lykeion]: 1 st , 2 nd grade). The 
team has just finished the enrichment of the book of 6 th grade of Primary school and is currently working on 
the books of 3 rd , 4 th , 5 th grade. 

On the eClass platform are also uploaded the aims and targets of each course per grade and per subject, as 
they are published in the national state Curricula (Government Paper, 2003: 303/13.03.2003, v. B' & 
Government Paper, 1998: 406/05.05.1998, v. B') as well as the Teacher’s Book for each course. 

Religion is an obligatory course in the greek educational system taught at both Primary (4 out of 6 
classes) and Secondary Education (3 years High School and 3 years upper high school [Lykeion], Its content 
contains information on history, practice, art, on religions and their role as way and expression of life 
(Christianity (mainly) and references to other religions e.g. Islam, Judaism, Buddhism, Hinduism etc). 

The interactive components/applets developed and embedded on the interactive books of Religion have a 
great variety in their formats and interactivity. More specifically, there are: (a) introductory presentations 
(presenting the basic points of the lesson (written and orally) (b) interactive maps and pictures (c) photo 
galleries, (d) videos, (e) links with Wikipedia, (f) links with a greek online Dictionary, (g) activities for 
consolidation, practice and/or feedback (quizzes, crosswords, puzzles), (h) music and songs (i) comics in 
books. 

For the development of the interactive activities for the course of Religion were used the following 
software programs: Articulate Studio 2009, Pro Adobe Flash Pro CS5, Wondershare, QuizCreator 4.1.0, 
Audacity, Colagedit, ispring free. Eclipse Crossword, Hot potatoes. Acrobat Reader, Comic lab. 


4. PEDAGOGICAL AND TEACHING APPROACH OF THE 
INTERACTIVE E-BOOKS FOR THE COURSE OF RELIGION 

The interactive schoolbook can be used in the teaching of the course of Religion with interactive whiteboard. 
The teaching of Religion course in schools is delivered till today in the traditional way, teaching in groups, 
accompanied with a power-point presentation prepared just for the purposes of the specific lesson. 
Occasionally, when the teacher is more familiar with computers, he may use educational software. Usually, 
though ICT is used in the teaching, just as visual medium without activating the students, because they still 
sit and watch and not participate actively. This is important in the course of Religion because it is a course 
based on experiences from life and the above described way of teaching does not exploit the possibilities of 
the computer in this direction. Also it does not help the students realize the interdisciplinarity of the course of 
Religion with other courses (e.g. History, Geography, Visual Arts). Among the advantages of teaching with 
such interactive books is the possibility offered to the student to participate actively during the lesson and 
make correlations and creative comparisons with either other lessons or other courses thus supporting the 
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interdisciplinary learning approach (Mitropoulou, 2011). The content uploaded on the interactive books of 
Religion is expected to offer multiple representations of the various information, which is envisaged to result 
to their better imprint in the students’ memory, as «the information is presented in a more attractive, direct, 
pleasant and comprehensible way» through interactive pictures, interactive maps, comics, applets etc. 

The introductory presentation created with Articulate software is a presentation with the main points of 
each lesson of the book (duration 3-5 min). The student can both see these points in text and listen (or not 
listen if he does not want) them at the same time. When possible, the text is accompanied by a relevant 
picture. The introductory presentation can be used by the teacher to introduce the lesson to the students 
(presenting the new information so that the students will use it as a base to construct their new knowledge) or 
for review at the end of the lesson (review or feedback, and consolidation of the information learnt) or before 
the next lesson to help the students recall in their memory relevant information and connect them with the 
new information that they will be taught, thus facilitating the assimilation and adjustment. The students on 
their part can review the main points of the lesson at their own place and time. 

The photo galleries present different views of a theme, e.g. photos of Mt Thavor or river Jordan, or icons 
of Transfiguration or Buddha. The photo-galleries developed for the interactive books, depending on the 
topic, they use photographs, byzantine icons, paintings of famous painters, etc. The students can select the 
picture they wish to observe in e.g. a carousel or scattered cards. By clicking on it once they can enlarge it 
and notice the details of it, e.g. posture or facial expression, and by clicking once again they can read details 
about it, e.g. if it is a famous painting, the students can get information on its title, artist, date, museum where 
it is etc. Thus the students can actively practice to observe the paintings and notice e.g. the common elements 
or persons that there are on them and their posture e.g. in Transfiguration, Jesus is dressed in white clothes 
and is inside a circle of light, on his right and left there are depicted Moses and prophet Elijah, while at the 
bottom of the painting there are fallen down his three students Peter, John and Jacob. The students are asked 
to observe that e.g. in all the paintings, the students are fallen on the ground, Jesus is in white, they are on a 
mountain (Mt Thavor) etc. This activity responds to the special needs of Religion painting, which are 
symbolic and therefore not understood. Another component which helps in this direction concerns the 
interactive icons. The student, when clicking with the mouse on a person of the icon, can get information 
about him/her e.g. who he is, what is the symbolism, why he is there. With the photo-galleries and the 
interactive pictures the students can actively see and understand why a theme is depicted in this specific way 
and which persons participate in this event (e.g. Baptism), or see the parts and objects inside/outside a sacred 
place (mosque, synagogue, hindu temple). This activity has a direct use for the course of Religion as it is 
related with its symbols, without which it is not possible to understand e.g. historical or art issues. This kind 
of activities help the students practice their observation skills and their critical thought, as they are involved 
actively in creative comparisons e.g. compare the religious symbolisms in the byzantine icons with the same 
symbolisms in the paintings of famous painters (e.g. El Greco) thus involving in interdisciplinary approach 
with the Visual Arts Course. 

The words which are links to the online modern greek dictionary (http://www.greek- 
language.gr/greekLang/modern_greek/tools/lexica/triantafyllides/index.html) are keywords, useful to help 
the students understand the content of the lesson as they also provide the religious meaning of the words, thus 
aiming to help the students become familiar with it and understand better the content of the lesson. When the 
students try to solve the crosswords they will need to know about the specific meaning a word has/acquires in 
the religious context. Because a word or phrase matching exercise like those in a crossword “represent the 
new knowledge with words, which -according to Bruner- consist ‘abstract symbol systems’ (symbolic 
representation of knowledge)”. (Samara, 2011, Mitropoulou, 2012) Additional information they can get from 
the links provided to the greek Wikipedia. 

The interactive maps can help the students gain consciousness of the time and place the event occurred 
(e.g. Exodus) and be able to realize the geographic location today (in the photo gallery they can even see how 
that place looks today e.g. desert of Sina). Also by clicking with the mouse on a world map they can see e.g. 
the countries (and their flags) where the death penalty is still applied). This is very important for the course of 
Religion, especially when it concerns historical events or issues (e.g. Edict of Milan, world map of religions 
today) as the students will realize that the historical events they are taught about are not fictitious and they do 
not concern a remote past but they are related with present situation (e.g. Israel). The students with Google 
earth can visit the cities e.g. that St. Paul visited during his journeys or the area that St Basil lived on a 
present map (from Google maps). It is envisaged that they will connect the time he lived with the present 
condition and gain consciousness of the time with a timeline cursor. 
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Finally, the quiz exercises, which are located at the end of the lesson, contain various behavioristic type 
exercises (true-false, multiple choice, word completion, correct order, word or phrase matching) that offer 
the students the possibility of self-evaluation, because the students can either get immediate feedback of their 
answers, or see again the exercises and the correct answers. Because, the closer the feedback and the correct 
answer is to the answer given by the student, the better and longer is imprint in the student’s memory. At the 
end of the quiz exercises, there is a final percentage of the correct answers given by the students. This is to 
provide (a) to the teacher the possibility to check if the students achieved the aim of the lesson and 
constructed their knowledge, and (b) to the students the chance to check if their new gained knowledge 
responds to their views up-to-day, thus improving their meta-cognitive abilities. 



Figure 1. Introductory presentation, interactive picture, photo gallery, interactive map, quiz 


The teacher’s role is that of facilitator, guidance to the journey of learning; he provides the students the 
suppliers, shows them the final destination (knowledge) but it is the students that will select how to use the 
supplies and which way to follow to get there. 


5. CONCLUSION 

The interactive schoolbook of Religion, enriches their content with multiple ways and offer the students 
multiple representation of the various information, resulting to their better imprint on the students’ memory. 
Thus, the information is presented in a more attractive, direct, pleasant and understandable way. It is 
expected that this interactive presentation of the learning content from the e-schoolbooks presents some 
advantages, e.g. provide the students the possibility to participate actively during the teaching approach and 
make correlations and comparisons with the content of other lessons of the same or other course thus, 
supporting the interdisciplinary learning approach. We envisage that this kind of presenting the content of 
Religion will create favorable conditions for the teaching approach of the course of Religion and it will be an 
initiative for renewal of its teaching strategies (Mitropoulou, 2008, 2011). We are planning a research in the 
secondary education schools of Greece to get feedback of the views of the teachers and the students when 
these interactive books are used during the teaching of the Religion course, the results of which will be 
presented at another conference next year. 
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ABSTRACT 

Life in the contemporary society and ongoing globalisation processes result in growing demands on educators, including 
language teachers in higher education. The frequently accentuated so-called postmethod approach to foreign language 
teaching gives teachers a lot of freedom and flexibility but also creates a large space for various types of challenges and 
dilemmas. These are related to issues ranging from the inevitability to perceive teaching in a broader educational context 
to the necessity to internalise the lifelong learning concept, bringing along the categories of formal, non-formal and 
informal learning. Besides, there is no doubt that language learning/teaching processes are to a great extent facilitated by 
the rapid development of ICT. This paper is aimed at explaining the authors’ beliefs related to the role of the teacher in 
curriculum development and the level of his/her control over the content of learning within the above specified context. 
The central part of the paper relies on examples illustrating the teaching and research activities carried out at the 
Language Centre of the University of Pardubice, Czech Republic, that are powered by the efforts to use elements of e- 
learning for bridging the gaps between formal and informal language education. 
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1. INTRODUCTION 

The demands life in the contemporary society makes on humans are unparalleled from the point of view of 
the history of mankind. They are a consequence of rapid development of modern technologies, primarily 
information and communication, the increasing pace of globalisation tendencies and unprecedented migration 
and travel worldwide for work or other reasons. For most people, this results in the necessity of learning at 
least one foreign language. Simultaneously, the requirements on foreign language teaching in higher 
education throughout Europe reflect the principal goal of the European education policy, namely preparation 
of future experts and specialists for effective functioning on the job market. Therefore, the efforts of higher 
education institutions are being driven by a well-justified plan to optimise conditions for the desirable shift 
“from expert students to novice professionals” (Reid et al. 2011) also in the area of complex language 
development. 

As the language that plays the role of modern lingua franca is English, top quality and efficiency are 
demanded in the area of English language teaching (ELT). Language teaching specialists can draw on 
numerous positives in current approaches to methodology in terms of what some authors call the postmethod 
stage (e.g. Kumaravadivelu 2008) which is characterised by a great extent of freedom and flexibility. There is 
no single generally accepted or prescribed “right” method, and teachers are encouraged to be inspired by 
whatever traditional or modern approaches in case they consider them appropriate, useful and efficient. This 
type of freedom and flexibility presupposes a high level of teacher cognition, confidence and autonomy, 
reflected experience and good decision-making skills. At the same time, it also requires a high degree of 
teacher awareness of a broader educational context and of the importance of the concept of lifelong learning. 

Establishing effective links between formal, non-formal and informal learning within the framework of 
life-long learning represents one of the biggest challenges at all levels of educational systems. It is mainly 
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due to the fact that institutionalised formal learning, non-formal learning occurring in formal learning 
environments but not formally recognised (e.g. workshops, interest based courses, conferences) and informal 
learning of rather incidental or random nature (e.g. CEDEFOP 2001) are perceived as complementary but at 
the same time, they bring about very different curricular implications. For example. Smith (2000) contrasts 
all of the above mentioned categories in order to present the notion of curriculum as the boundary between 
formal and informal education, thus initially questioning the relevance of curricular theory and practice 
within the area of informal education but finally concluding about the necessity to conceptualise even the 
area of informal learning in terms of the language of curriculum (e.g. informal programmes or projects 
instead of formal courses at school). The importance of searching for “bridges” between formal, non-formal 
and informal learning within the particular educational contexts is also obvious from the perspective of 
addressing all domains of general aims of education, since according to Rabusicova (2008) formal learning 
has the highest potential in the area of cognitive development, non-formal learning in the area of the 
development of skills, and informal learning in the affective domain. Translated into the pedagogical 
discourse of language education at the tertiary level, this triad might be interpreted in terms of the 
complementarity of formal language courses at school, opportunities for developing language skills non- 
formally in an authentic or simulated working context and informal opportunities to practice a foreign 
language in a non-threatening context, thus building positive attitudes towards the language as a means of 
communication. 

Besides, one should not neglect the obvious fact that the processes of learning foreign languages are 
nowadays to a large extent facilitated by means of ICT. The use of ICT for language teaching/learning 
purposes undoubtedly represents a rapidly developing area, yet it might be still useful to categorise the 
teacher roles according to a traditional criterion of the degree of control over the content of learning. At one 
end of the continuum, we may observe teachers with a high degree of control over the content of learning 
within formal Moodle courses, while at its other end teachers deliberately resign on their controlling role and 
draw on the potential of informal learning within widely used social networks, such as for example 
Facebook. However, searching for effective bridges between formal, non-formal and informal language 
learning in the area of tertiary education presupposes overcoming the above mentioned unilateral views and 
implies the need for a more balanced approach taking into the consideration the complexity of language 
development (cognition, skills and attitudes) and the need for individual learning paths. 


2. BRIDGING THE GAPS BETWEEN FORMAL AND INFORMAL 
LANGUAGE LEARNING 

2.1 Specifying the Context 

With regard to the above mentioned issues, we will present several examples of efforts carried out at the 
Fanguage Centre (LC) of the University of Pardubice, Czech Republic. Many of the activities the FC staff 
concentrate on are related to EU-funded projects. One of them, EFPiPL or European Fanguage Portfolio on 
Student’s Journey through Studies into Professional Fife, was an international project involving secondary 
and tertiary education institutions from six EU countries and running from 2009 till 2011. It was aimed 
primarily at developing teaching methods and materials in cooperation with the world of work, enhancing 
learner autonomy and promoting plurilingualism and lifelong learning. Another project, UNICOM - 
Innovative and Integrated Concept of Fanguage Training towards Quality Assurance, Excellence and 
Internationalisation, has been running since January 2012 and will be completed by the end of 2014. Its 
outcomes are expected not only to enhance the quality of learning primarily in the area of Fanguages for 
Specific or Academic Purposes (LSP, FAP) in the formal higher-education setting but also to motivate the 
students to increase the share of informal learning in their own education, emphasising the benefits it can 
yield for life-long learning. 
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2.2 Examples of Bridges between Formal and Informal Learning in Higher 
Education 

2.2.1 ELPiPL Blogging 

Using portfolio assignments generally represents a significant innovation of language teaching courses, 
mainly in terms of the development of learner autonomy. Within the ELPiPL project, this aim was pursued 
not only by means of the tasks from the printed portfolio brochures but also with the help of the social 
networking platform NING. For this particular purpose, the NING platform was available only for authorised 
users after an invitation by the NING administrator of the project. The blogging part of the ELPiPL project in 
the Czech educational context proved that even lower-level students (A2) were able to formulate successfully 
blog posts mainly of a reflective nature (e.g. personal stories “How I started learning foreign languages”, 
“Why I find learning foreign languages important”) and that they demonstrated a high-level of peer support 
by means of commenting positively on the accomplishments of the others, thus building positive attitudes 
towards the use of social networking for one’s own language learning purposes. 

2.2.2 Progress Test 

An example of an effective bridge between formal and informal learning is Progress Test administered as 
described below. Following this particular procedure, the test had a dual function: (a) an e-learning/testing 
tool in the formal setting of an LSP course in higher education, b) a tool supporting learning from home or 
another informal setting. 

This test was used with a total of 48 students of English for IT. They had to take the test twice within four 
months. The test was always active for a specified period of time and the students could choose when and 
where to take it. They were asked to write a short comment on their results, including a note on the difference 
in their two scores, and encouraged to include other comments. The purpose of this assignment was twofold. 
First of all, the test was intended to provide the students with feedback on their progress in English; this 
information was also important for the teacher. Secondly, the test results and student comments provided the 
teacher with data for content analysis focused on finding answers to the following questions: (1) How many 
students made progress? (2) What reasons do the students provide for their progress/decline? (3) Can an 
assignment of this type have an influence on the students’ self-assessment and motivation? 

As this paper does not provide enough space for a detailed data analysis, let us summarise the most 
important results. This preliminary research will be further extended and the full results will be published 
later. The answer to question ( 1) is displayed in figure 1 below. 


■ improvement 
decline 

25,5 2% ■ no change 

■ no/mcomp arable 
answer 


Figure 1 . Progress Test - improvement 

A coding technique was used for the analysis of the students’ open comments, which provided data of 
qualitative character. The most relevant answers to question (2) can be summarised as follows: The most 
frequented reasons the students gave for their improvement was that Test 02 had seemed easier to them. 
Some said they had been lucky, a few claimed they had been studying hard. Several students gave watching 
films in English as a reason for their improvement, a small number said they had had better class attendance. 

Although the students were not asked to comment specifically on their learning processes or motivation, 
the analysis of their open comments yielded a definitively positive answer to question (3). Several students 
said that having seen their scores, they wanted to improve their English. Some of them expressed 
(dis)satisfaction with their results, which showed the tests and the assignment had given them an opportunity 
for self-assessment and self-reflection. 
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2.2.3 Internet Forums 

Another example of a bridge between formal and informal learning is a web-based assignment. The students 
of a B1 course of English for IT were assigned the following task: Visit an internet forum on (an) operating 
system(s) and write replies to two different posts. The teacher found inspiration for this type of assignment at 
an interview she had carried out with an expert working in the IT business within her research incorporating 
the analysis of the communicative needs in English for future graduates of the IT programme. Among other 
tips, the interviewee advised to visit internet forums on specific topics to get acquainted with the language IT 
experts use to communicate about issues related to their profession. In this context, internet forums can be 
considered informal learning settings as they represent platforms on which not only experts from practice, but 
also students exchange their expertise, experience and ideas in real-life communication outside the education 
system. Apart from this, the students’ posts contributed to the bank of the most frequented errors that could 
be analysed and worked with in classes. 

2.2.4 Online Games as an Extension of Formal Language Training 

Undoubtedly, the internet represents an enormous space for spontaneous formation of professionally-related 
groups (see above the IT forums) but also of the communities whose members share the ultimate goal of 
“mere” game-playing. As one of the components of formal language courses is training in presentation skills 
in order to prepare the future professionals for the task of giving well-targeted and persuasive work-related 
presentations, one of the model performances was given also by the teacher. This model presentation was 
carried out with dual aims: demonstrating particular language and non-linguistic means for giving a business 
presentation, and at the same time inviting the students to join an online game where they would practise 
English with their teacher after the end of the formal language course and within the specific community of 
players. In 2012, the unique context of the European football championship was taken advantage of, and the 
students were invited to play the so called fantasy football together with their teacher. 


3. CONCLUSION 

The assignment and activities presented above support learning theory suggesting that learning is promoted 
or enhanced when students are actively involved in the learning, when assignments reflect real-life contexts 
and experiences and when critical thinking or deep learning is promoted through applied and reflective 
activities (Smart and Cappel 2006). From this perspective, it is definitely worth to search for and analyse 
potential bridges between formal and informal language learning systematically and on a long-term basis. 
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ABSTRACT 

Learning to learn has been identified as a key educational competence. Over the next two years, as part of the INSTALL 1 
project, NUI Maynooth is testing the effectiveness of an exploratory group technique, the Narrative Mediation Path 
(NMP). which has been developed to promote reflective thinking skills. To date, interviews have been conducted with 
200 first year students. Common themes emerging from these interviews regarding student engagement are discussed. 
Notable was the number of concerns relating to coping with academic demands. From these 200 students, 20 students 
will participate in the subsequent phases of the INSTALL project utilising the NMP. 


KEYWORDS 

Learning to Learn, Underachievement, Higher Education 


1. INTRODUCTION 

The EU2020 strategic objective of a more cohesive growth through knowledge based economy calls for 
improved models to sustain the acquisition of the key competence, learning to learn, for disadvantaged and 
underachieving Higher Education students. The National University of Ireland Maynooth (NUI Maynooth) in 
collaboration with its European partners, INSTALL, is responding to this challenge by exploring innovative 
solutions to address students’ constraints to acquiring and developing the learning to learn competence. 

1.1 Learning to learn 

The transition from second level instruction to university education involves students adapting to arguably a 
different type of learning. Traditionally within the university domain greater emphasis has been placed on 
self-initiated study. Consequently, for many students the challenge becomes one of learning how to learn. 

The term learning to learn refers not to a unitary skill, but rather a range of skills identified as 
contributing to optimal learning, including cognitive skills, motivation, organisation of time, and 
self-regulation (see Amalathas, 2010, for further details). Within the literature, there is a particular focus on 
the important role played by metacognitive skills in learning to learn. Findings have shown that university 
students who score highly on measures of metacognitive skills tend to perform better on academic tasks (e.g., 
Van der Stel & Veenman, 2010). Reflection is one of the core metacognitive skills (Lew & Schmidt, 201 1). 

In terms of the INSTALL project, an exploratory procedure known as the Narrative Mediation Path 
(NMP) has been developed to support the ability of students to reflect on their university experiences. 


INSTALL, Innovative Solutions to Acquire Learning to Learn, is a European funded project (Erasmus Multilateral Projects, n° 
517750-LLP-l-IT-ERASMUS-ESIN). Partners involved are: University of Naples, Italy, National School of Political and 
Administrative Studies (NSPAS), Romania; University of Aarhus, Denmark; NUI Maynooth, Ireland and University of Seville, Spain. 
Please note that this project has been funded with support from the European Commission. This paper reflects the views only of the 
authors and the Commission cannot be held responsible for any use which may be made of the information contained therein. 
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Specifically, this tool was designed for use with students following semesters during which they have 
struggled with academic activities. The NMP is a group narrative tool. Therefore, the project builds on the 
more recent research in this area which has investigated the promotion of metacognitive skills through social 
interactions with peers (e.g., De Backer, Van Keer & Valcke, 2012). 

The current paper focuses on the research undertaken with students enrolled at NUI Maynooth and also 
discusses the research that will be conducted throughout the coming months as part of the INSTALL project. 

1.2 Background 

NUI Maynooth has expanded rapidly with an increase of 21% between 2004/05 and 2011/12 in the total 
number of students, to become one of the fastest growing universities in Ireland. Approximately one quarter 
of the students come from non-traditional backgrounds. In addition, NUI Maynooth attracts more students 
whose parents did not go to Higher Education than other universities (Irish University Study, 2009). 

1.3 Statement of the Problem 

Twenty-first century university students are expected to be flexible, digitally competent, self-regulated, 
learners. To succeed, students must be ready to respond quickly to ever-changing intellectual and 
technological environments. Evidence would suggest, however, that students entering university directly 
from second level are ill-prepared, making the transition without the generic skills needed to cope at that 
level (Hyland, 2011: 16). Professor Tom Collins chairman of the NCCA voiced his criticism as follows: 

There is palpable concern in Higher Education regarding the capabilities and dispositions of 
students entering [university] straight from second level. The manner in which the points 
system rewards rote learning, instrumental learning and memorisation while simultaneously 
discouraging exploration, self-directed learning and critical thinking means that even relatively 
high achieving second-level students can struggle on entering third-level' (Collins: 2010). 

The selection of school leavers for third level education in Ireland is founded on a points system based on 
an applicant’s results in six subjects in the Leaving Certificate examination. The Leaving Certificate is a high 
stake examination, and because of this, the backwash effect on teaching, learning and the student experience 
is considerable (ibid: 4). The race to optimise points greatly influences the basic level of knowledge, 
cognitive skills and learning habits of students (Hyland 2011: 4). Concerns have been raised by policy 
makers and academics that the Leaving Certificate rewards rote -learning rather than problem-solving, critical 
thinking, or self-directed learning (ibid: 16). Such concerns about students not being adequately prepared for 
Higher Education are not unique to Ireland. Similar concerns have been voiced in the US, UK and in other 
EU countries (Hyland 2011, Gogus and Arikan, 2008: 279). Of course, there are also many other factors 
which influence student underachievement and non-progression, including social, cultural, financial, and 
institutional factors. The problem seems to be that most universities have not yet been able to translate what 
[they] know about student lack of ‘preparedness’ ‘into forms of action that have led to substantial gains in 
student persistence and graduation’ (Tinto, 2006: 5). 

As part of the INSTALL project, NUI Maynooth is responding to this challenge by developing a tool, the 
NMP, to help students acquire skills related to learning to learn. 


2. METHODOLOGY 

2.1 Phase One: Interviews with Students 

Between September 2011 to May 2012, 200 face-to-face interviews took place with first year students who 
visited the Academic Advisory Office (AAO) in NUI Maynooth. For each student, a written record was kept 
to detail the reason for the visit. Analysis of the written records was undertaken by an independent coder to 
identify key themes to emerge from the interviews. 
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2.2 Phase Two: Implementation of the Narrative Mediation Path 


Participation in the remaining two phases of the project requires fulfillment of a number of set criteria as 
shown in Table 1. All students will be briefed about the project and informed consent will be obtained. 

Table 1. Student selection criteria. 


Category A 
Academic Criteria 

Category B 

Social and Economic Criteria 

Category C 
Preferential Criteria 

• Passed at least one 

• Ethnic minority. 

• Achieved one high 

module by 

• Disability. 

grade in the first or 

compensation 2 . 

• In receipt of a Local 

second semester of first 

• Failed at least one 

Authority Grant. 

year of undergraduate 

exam in January or 

• First generation 

degree programme. 

May and repeated it in 
August (and then 
passed). 

• Self reporting - may 
pass exams but do not 
have a score above 
50%. 

university student. 

• Scored at least 375 
points in the Leaving 
Certificate. 


Note. Students must satisfy at least one of the criteria from Category A or Category B. 


There will be two cycles of the NMP programme. Initially, the first sample of 20 students will commence 
the programme in October 2012. A second sample of 20 students will begin in February 2013. In the 
intervening period between the two cycles, refinements will be made to the programme. Prior to the start of 
the programme, students will answer open-ended questions related to their university experiences to obtain a 
preliminary measure of their entry reflective skills. Subsequently, the written responses will be evaluated 
according to the Reflective Functioning Scale devised by Fonagy, Target, Steele and Steele (1998). 

The NMP programme will be presented to students across six meetings. Each meeting is guided by a 
trained instructor. There are four modules to be covered as follows: (i) metaphorical; (ii) iconographic; (iii) 
writing; (iv) bodily. A description of the contents of these modules can be found in Freda, Esposito, Martino 
and Monteagudo (2012). The modules are completed in a set order, with the sequence of the four modules 
arranged to encourage the cognitive, social, and emotional involvement of all the students. It is proposed that 
such an involvement is made possible by using narrative stimuli (e.g., vignettes, written accounts) and 
procedures designed to promote and develop reflective thinking skills. 

To assess whether there has been any change in reflective skills, students will again provide written 
responses to a set of open-ended questions. Responses will be scored using the Reflective Functioning Scale. 

2.3 Phase Three: Tracking Academic Progress 

Six months following completion of training, the academic performance of each student will be monitored by 
examining the grades obtained by students for their chosen second year degree modules. 


3. PHASE ONE RESULTS 

The 200 written reports were reviewed to identify the main reason for the student’s visit. Fourteen categories 
were established as summarised in Table 2. 


2 Students may pass by compensation by passing two of their subjects, obtaining at least 35% in their third subject, and obtaining at least 
40% on aggregate, provided that the two passed subjects are compatible in second year. Such students may only continue in second year 
with the subjects in which he or she has obtained at least 40%, hence students cannot pass by compensation in a subject that is a 
requirement in a second year programme. 
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Table 2. Categories employed to code the reports of the interviews conducted by the Academic Advisory Office. 


Category 

Brief description 

Onwards referral 

Student was recommended to contact the relevant 
university administrative department (e.g.. Admissions) 
or academic department (e.g., Psychology), given the 
specific nature of the query. 

Dislike of course 

Student explicitly states a lack of enjoyment for the 
chosen degree subjects. 

Failure to meet course requirements 

A student who may not have submitted an essay or 
assignment; who has not attended lectures or tutorials; 
who has not achieved the required mark in a module. 

Mental Health Issue 

Any mental health issue (e.g., depression, anxiety), 
often accompanied by medical certification. 

Physical Health Issue 

Any physical health issue (e.g., illness, treatment, 
surgery, injury) often accompanied by medical 
certification. 

Financial 

Issues with grants, funding, debt. 

Family 

Bereavement, family illness, unemployment, childcare. 

Commuting 

Living too far away from the university which is 
affecting attendance, study. 

Finding course difficult 

Student explicitly states that the course is challenging. 

Problems with studying 

Difficulties with writing essays, knowing what material 
to study, compiling notes in lectures. 

Time management 

Difficulties in planning revision, study time, reading 
time. 

Desire to change course 

Student expresses a wish to study a different subject. 

Transfer to another university 

Student is considering leaving to pursue a programme 
in a different third level institution. 

Other 

Any other reasons which cannot be coded. 


Figure 1 shows the number of students attending the AAO across the 14 categories. The most common 
reason for students to contact the AAO was to change their course. However, approximately 35% of the 200 
students recounted problems explicitly linked to their chosen courses. Twenty-six students were concerned 
about a failure to meet course requirements. Other students needed assistance because they did not like their 
course (22 students) or were finding the course difficult (17 students). 



Figure 1. Reasons for first year students attending the Academic Advisory Office. 
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4. CONCLUSION 

Student disengagement and academic underperformance is complex, although it is generally recognised as a 
path that can be traced, with problems that can be resolved through enhanced student support (Thomas 2005: 
63, Mann, 2008). Student support services, such as the AAO at NUI Maynooth, can help to directly enhance 
academic supports for at risk students. Indeed, the initial phase of this research has helped to shed further 
insight into some of the most widespread difficulties experienced by first year students attending NUI 
Maynooth. A visible trend was that students expressed concerns specifically linked to their academic 
performance, for instance, about not fulfilling course requirements or finding the course difficult. Our goal 
now is to investigate potential ways to support these students. Helping students to acquire key skills such as 
the learning to learn competence offers one such possibility. Through participation in the INSTALL project, 
the aim is to test the effectiveness of the NMP in developing reflective skills. The ability to constructively 
think back on past experiences and to recognise the potential for change can all be categorised as reflective 
skills; skills that may be highly valuable for students attempting to successfully complete university. 
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ABSTRACT 

Ontologies describe a body of knowledge and give formal structure to a domain by describing concepts and their 
relationships. The construction of an ontology provides an opportunity to develop a shared understanding and a consistent 
vocabulary to be used for a given activity. This paper describes the construction of an ontology for an area of 
mathematical logic as part of an EU Lifelong Learning project and on-going work exploring how this ontology has been 
used to inform the development of learning resources and appropriate structuring mechanisms. Finally, the paper 
discusses future work developing enhancements to a virtual learning environment to allow better integration between 
domain ontologies and on-line courses. 
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1. INTRODUCTION 

The on-line delivery of learning materials is rapidly becoming an important tool in school and university 
education programmes. Virtual learning environments afford new opportunities in course sUuctures and 
delivery formats. Pedagogical practice is undergoing a transition as new delivery mechanisms are integrated 
into existing course structures. These learning technologies provide interesting opportunities to develop new 
strategies for the delivery of learning materials to students. Many universities and schools now use virtual 
learning environments (VLEs) such as Moodle (Dougiamas and Taylor, 2003) and Blackboard (Bradford et. 
al, 2007) as a mechanism to collect together learning activities. However, the adaptive functionality in these 
environments can only be exploited if sufficient information is known about domain to develop suitably 
interesting learning paths. 

In an era of increasing school and university class sizes, identifying learning paths for individual students 
can be challenging due to time and resource consUaints. The authors are part of the MALog project, an EU- 
funded project that is working to develop open learning resources to improve mathematical logic knowledge 
and skills. Mathematical logic is an umbrella term covering a branch of mathematics that includes topics 
such as set theory, propositional logic, predicate logic and proof. These topics are seen as fundamental to 
developing knowledge and skills used in many scientific, mathematical and engineering disciplines. Yet 
developing resources suitable for delivery in the wide variety of learning environments and educational 
systems found throughout Europe presents some difficulties. The problems include identifying key concepts 
and their relationships, identifying the specific skills required, and recognising the differences in terminology 
and educational approaches. Collecting and recording this information in a useful format is a first step to 
developing better learning resources. 
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Figure 1. Ontology classes representing set theory topics. 

Building an ontology of a knowledge domain allows a shared representation of concepts and their 
relationships to be explored. When the knowledge domain (in this case mathematical logic) is described with 
an appropriate level of detail, the information can be useful enough to inform the structure and content of 
appropriate learning resources. Furthermore, the information gathered can be used to develop learning paths 
suitable for students with a range of abilities and educational backgrounds. The approach taken here has 
been to develop an ontology of mathematical logic which identifies key mathematical concepts, mathematical 
topic areas and prerequisite information. 

The aim of this project is to use the information gathered in the ontology of mathematical logic to develop 
a suitable course structure. Assessing existing knowledge and providing suitable remedial support is vital to 
successful learning (Fensham, 1972). An individual learning path will be appropriate to the needs of the 
student by accounting for their individual learning style and rate of progress. By identifying problematic 
concepts, the adaptive functionality found in many VLEs can be exploited to produce learning paths where 
material is automatically recommended to students based on their progress through learning materials. This 
paper describes the construction of an ontology in the area of mathematical logic and describes on-going 
work that uses this ontology to develop and structure educational resources appropriate for a range of 
students. 


2. ONTOLOGIES AND LEARNING 

An ontology provides a mechanism to formally represent a body of knowledge. One of the earliest 
definitions of ontologies (as used in information science) is the “specification of a conceptualization” 
(Gruber, 1995). Ontologies have since developed into an important technology of the Semantic Web 
(Berners-Lee, 2001), adding machine recognisable meta-data to information available on the World Wide 
Web and providing new opportunities in data processing, sharing and reasoning. Technologies such as RDF 
and the Web Ontology Language (OWL) have supported the development of the Semantic Web and 
emphasised the importance of the ontology concept. Outside information science, the medical science 
community have developed numerous examples of ontologies covering human, plant and animal biology 
demonstrating how large vocabularies can be successfully developed, used and further explored. 

In a learning context, an ontology can be used to formally describe a set of concepts (or topics) and the 
relationships between these concepts. A domain ontology can be used to describe an agreed organisation of 
concepts and a shared vocabulary which can be used to ensure consistent use of technical terms. An 
ontology can provide students with a mediating artefact to aid understanding of the structure and vocabulary 
of a domain. Henze et. al. (2004) demonstrate how ontologies can be used to personalise e-Learning material 
and recent work by Laroussi (2012) describes how ontologies can be used to model user behaviour in 
adaptive learning environments. The MALog project has been developing an ontology to provide a 
description of the topics in the mathematical sub-field of mathematical logic. The ontology has been 
constructed from a detailed survey of mathematical logic textbooks, Internet resources and other material to 
identify specific concepts and topics. Key words and phrases have been identified and organised into 
categories, first more general and then categories with increasing specificity. High level categories became 
entities within the ontology class hierarchy (e.g. 'set theory', 'axiomatic set theory') and specific concepts 
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within those areas have been created as individuals within the ontology and classified under one or more 
ontology classes. Figure 1 shows an example of this hierarchy. As part of the process, information about the 
relationships between specific concepts has also been recorded within the ontology. Related topics are 
linked, similar concepts from different sub-fields are identified and mathematical symbols are connected to 
concepts. Production of the ontology is the result of a negotiation of meaning of the domain of mathematical 
logic. Further information on the strategy used for developing the ontology is described in Boyatt and loy 
( 2010 ). 


3. ONTOLOGIES AND LEARNING MATERIAL DEVELOPMENT 

The construction of the ontology is motivated by the desire to identify the mathematical logic topics and 
relationships between these topics suitable for inclusion in a set of related learning materials. The ontology is 
used as the semantic foundation giving an initial structure to the development of learning materials. The 
initial version of the ontology identified over 500 concepts (as ontology individuals) and 60 categories and 
sub-categories (recorded as ontology classes). The ontology information was then used to develop units of 
material on mathematical logic topics. The categories provided learning material authors with initial 
suggestions for how to structure units of material. The concepts (individuals) within each category (class) 
immediately provides a list of concepts to be included within a unit of learning material. For example, the 
‘NaiveSetTheory' class included concepts such as ‘Set_Union’, ‘Setjhitersection and ‘Set_Membership’ 
suggesting a unit of material covering naive set theory. For many topics it was easy to identify specific 
concepts but for some topics this process exposed differences in educational systems for how concepts 
should be organised and related. Explicitly enumerating mathematical concepts in the ontology also ensured 
concepts were not missed or duplicated inappropriately in the learning materials. Additional information 
stored such as related concepts and mathematical symbols could also be accessed by the material authors. 

The MALog project is a collaborative project between several European countries requiring the 
development of material in five different languages. Ensuring consistent use of mathematical terminology 
has been aided considerably by the shared vocabulary and representation of knowledge stored in the 
ontology. The shared artefact has ensured that learning material has been developed with consistent use of 
terminology. Consistent use of terms has reduced the possibility of learners becoming confused with similar 
mathematical concepts. Recording the reuse of terms has also allowed us to provide navigation aids to 
learners by allowing them to identify all instances of specific concepts, e.g. allowing a learner to find all 
instances of the concept ’Cardinality’ within the material. This mechanism has been provided by developing 
software to read the OWL ontology file and display information within Moodle listing concepts related to the 
current activity. 

Other mathematical ontologies such as GeoSkills ontology developed by the I2Geo project identified 
specific competencies that learners can hold (Libbrecht et. al, 2008). These competencies connect together 
different topics to represent a particular understanding that should be held by the learner. This approach has 
been adopted for the mathematical logic ontology identifying competencies for three categories of learner: 
high-school students, university undergraduate students and industry workers. Over 100 competencies have 
been identified for each of these class of learners and are represented in the ontology in the ’Comptency’ 
class. For example, the ’Create_truth_table_for_NAND’ relates the concepts (ontology individuals) for 
’AND’ and ’NOT’. For the learner material author, the competencies provide a structure to units of learning 
materials and help build a basic structure of conceptual prerequisites. We are developing software that reads 
the entities from the OWL ontology file (using the Apache Jena API) and creates activities within Moodle 
that relate to each competency. Activities are ordered appropriately within the environment (using the 
limited adaptive functionality available) to ensure that students encounter material about ’AND’ and ’NOT’ 
before learning specifically about ’NAND’. The student using the ontology information can use the 
information to identify their attainment level and the competencies they should understand. Quizzes that test 
specific competencies are created ready for learning material authors to insert suitable questions that would 
enable the next set of material within the environment. 
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4. FUTURE WORK 

This paper describes work in progress developing and using an ontology of mathematical logic to produce 
learning materials and structure this material in a manner appropriate for use by students in schools, 
universities and industry. Undoubtedly the ontology of mathematical logic will continue to develop and be 
enhanced as it is used to inform the development of further learning materials. We expect the ontology to be 
enhanced with additional vocabulary, additional languages and cultural terms using strategies such as those 
developed by Espinoza (2008) and new connections formed between topics. Nevertheless, the ontology has 
reached a sufficient level of maturity to be useful to practitioners. The current approach is for the ontology to 
inform the construction of courses within our virtual learning environment. Courses are mostly manually 
constructed using the information gathered ontology editor view of the ontology (see Figure 1). On-going 
work is to further develop plug-in functionality for the Moodle VLE to create a basic course structure. We 
intend for activities in Moodle to be related to specific competency individuals from the ontology and 
learning materials to be related to specific topic individuals. 


5. CONCLUSION 

This paper has described a strategy for developing learning materials and structuring the delivery of that 
material using information collected in an ontology. While the work described is on-going, it shows the 
advantages of collecting domain information with an ontology to agree consistent use of vocabulary across a 
project involving multiple languages and providing a structure to develop learning resources appropriate for a 
broad range of learning environments. 
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ABSTRACT 

Technological developments have altered pedagogies in classroom teaching but approaches to teacher professional 
development have remained largely unchanged. The purpose of this paper is to describe an evolving learning process that 
spans the last decade and draws from three different investigations into professional development. The author compares 
and contrasts the key findings from two independent studies on teacher professional development. One study was situated 
in a face-to-face action research context and the other in an online environment. The conclusions drawn about the nature 
of effective professional development in these independent studies show that whether in online or face-to-face contexts 
educational developers need to consider similar elements. A review of the literature indicated that further research was 
required to enhance the development, implementation and evaluation of online professional development opportunities to 
meet the in-time learning needs of teachers in sustainable ways. The author concludes by summing up the advantages and 
disadvantages of moving towards online teacher professional development and considers possible applications. 


KEYWORDS 

Teacher professional development, informal learning, online learning 


1. INTRODUCTION 

Over the past decade technological developments have altered pedagogies in classroom teaching. Even so, 
approaches to teacher professional development have remained largely unchanged and typically delivered in 
a face-to-face mode (Lloyd & Duncan 2010). Problematically, face-to-face professional development has 
often been delivered as one-off workshops off site, whereas best practice models recommend embedded 
learning tailored to meet individual needs in the workplace (Bolt 2003, 2009; Guskey 2000; Lloyd et al. 
2005; Zepeda 2012). Problems with formal professional development programs are well documented but 
successful alternatives are less well publicized (Guskey 2000). 

The purpose of this paper is to provide a quick overview of an evolving learning process which spans the 
last decade and draws from three different investigations into professional development. The author first 
investigated teacher professional development in the context of a funded action research Quality Teacher 
Program in Western Australia (Bolt 2003, 2007). On the other side of the country Lloyd et al. (2005) 
investigated teacher professional development in the use of information communication technologies. 
Subsequently, the author investigated professional learning in the workplace in a study conducted in three 
large dynamic organizations (Bolt 2009ab, 2010). The first two studies investigated teacher professional 
development in separate, diverse contexts and resulted in the conceptualization of similar yet different 
models of professional development. In an unrelated context, the author conducted the third study to 
investigate the relationship between professional development and organizational change and development 
by considering adult learning in the workplace as a result of formal programs, non-formal programs, informal 
learning and incidental learning. Key findings indicated that adults in the workplace learned as much if not 
more through informal learning, yet managers and human resource developers rarely planned for it (Bolt 
2009ab, 2010). 

This paper contributes to current knowledge and practice about professional development. By comparing 
and contrasting findings from the three studies, the author considers ways in which formal and informal 
learning could be blended through the use of information technology to enhance teacher professional 
development. This paper may be helpful to academic developers, educational leaders and human resource 
developers who are interested in knowing more about models of professional development suitable for either 
online, blended or face-to-face scenarios. 
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2. A TALE OF TWO STUDIES 

The two studies undertaken independently in different locations, times and contexts arrived at remarkably 
similar conclusions (Bolt 2003; Lloyd et al. 2005). The Bolt (2003) study investigated the significant action 
learning experiences of teachers leading to professional growth. In the Bolt (2003) study ten case studies 
written by teachers participating in action research projects funded through the Quality Teacher Program over 
a one year period were analyzed and emergent themes identified. Although some of the participating teachers 
used technology in these projects the professional development focus was on action research in diverse 
situations. The Lloyd et al. (2005:1) study was commissioned by Queensland Society for Information 
Technology in Education (QSITE) “to inform its development of a position statement on professional 
development for teachers in the curricular use of ICT”. It was a small scale study in which conference 
delegates were surveyed and a range of educational leaders were interviewed (Lloyd et al. 2005). 
Interestingly, the resultant conceptualizations of professional development were similar (Bolt 2003, 2007; 
Lloyd et al. 2005). In Table 1 the similarities and differences between the two studies are compared and 
contrasted; then described in the following paragraphs. 


Table 1. Comparison of key elements of two models of teacher professional development 


Framework for improving teaching and learning 
through action learning (Bolt 2003) 

Elements of effective professional development (Lloyd 
et al. 2005) 

Context 

Context 

Collaboration 

Community 

Process (professional development + action learning) 

Theory and practice 

Endurance 

Time 

Professional growth (teaching and learning) 

Personal growth (ICT) 


2.1 Context 

Context is an important element of professional development in both face-to-face and online professional 
development. In the Bolt (2003) study teachers appreciated opportunities to solve issues of concern to them, 
with peers in school-based action research groups supported by organizational sttuctures and infrastructure to 
provide resources and time. By contrast, top-down approaches, teacher isolation, and lack of time were 
identified as constraining forces. In the Lloyd et al. (2005) study effective professional development was 
fostered in contexts in which learning was relevant, meaningful, practical and designed to meet immediate 
and ongoing needs (Lloyd & Duncan-Howell 2010). Action research was an important element in both 
studies and known to be an effective form of professional development (Bolt, 2003, 2007; Creswell 2008; 
Lloyd and Duncan-Howell, 2010; Zepeda, 2012). To sum up, an enabling context is important regardless of 
whether the professional development occurs in a linear program or an iterative online network. 

2.2 Collaboration and Community 

Whether in online, blended or face-to-face contexts there must be opportunities for participants to collaborate 
within communities of practice. Lloyd and Duncan-Howell (2010:70) described community as 
“collaborations during and following the professional development event and to ongoing connections and 
collaborations within local and extended communities”. Bolt (2003:140) noted “collaboration is the conduit 
through which professional growth flows and is fundamental to developing a learning organization . . . and is 
vital to developing a learning community”. In both studies collaboration and community were interconnected 
and involved action and reflection. In the Bolt (2003) study collaboration and community typically occurred 
in school-based groups of teachers in face-to-face situations as they engaged with the iterative processes of 
action research. In the Lloyd et al. (2005) study collaboration and community typically occurred online. In 
either case, collaboration with others in communities led to participants’ engagement with professional 
development as a process rather than as an event. 
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2.3 Theory and Practice 

Professional development must provide participants with opportunities to learn and subsequently transfer that 
learning into relevant workplace situations. In the Bolt (2003) study ‘theory' was referred to as a broad range 
of professional development inputs in relation to school-based action research projects. Lloyd and Duncan- 
Howell (2010) describe theory and practice in ways consistent with action research - iterative, linked to 
action and reflection, community, context and growth. Action research supports transfer of learning 
(Caffarella, 2002). 

2.4 Just-in-time and Sustainable-Over-Time 

Time is one major constraining force acting against successful implementation of professional development 
programs. Action research is timely, focusing on real life immediate problem-based issues. One of the key 
success factors identified in the Bolt (2003) study was ‘endurance’ - supported by the preceding elements of 
effective professional development participants sustained their engagement with school-based action research 
projects over a one year period to achieve their goals and answer personally relevant questions. Teachers 
directed their own learning and scheduled relevant associated action and reflection (Bolt, 2003). The 
professional development described in Bolt (2003) is consistent with Lloyd and Duncan-HowelFs (2010:70) 
“dual understandings of time” and depiction of effective professional development. Lloyd and Duncan- 
Howell (2010) link ‘in-time’ learning with practice and ‘over-time’ to theory as a result of reflection. 

2.5 Growth and Development 

In the Bolt (2003:155) study, analysis of ten cases identified “professional growth was evidenced by change 
at the classroom level in teachers’ pedagogy and student outcomes”. Lloyd and Duncan-Howell (2010) note 
the importance of increased personal and corporate knowledge and the ability of online communities to 
connect participants with ‘experts’ beyond their immediate situation. 

Lloyd and Duncan-Howell (2010) refer to online professional development metaphorically, depicting it as 
a Celtic knot rather than traditionally as a ‘journey’. Poignantly, they note that if professional development is 
a journey then for teachers in the 21 st Century “it is unlikely there is a destination” (Lloyd and Duncan- 
Howell 2010:73). Recent rapid technological changes have shortened the shelf life of knowledge, resulting in 
dynamic environments and increasing the need for lifelong learning, responsibly shared by individuals and 
organizations (Burns 2002). Consequently, there is increased need to learn informally ‘on-the-go’. 


3. THIRD STUDY SHOWS IMPORTANCE OF INFORMAL LEARNING 

In the previous two studies participants were teachers and educational administrators. In the third study 
participants were adult learners in the workplace many of whom were providers of training and professional 
development. Data were collected through 70 semi-structured interviews and 210 responses to a 23 question 
survey. Overall, participants in this study rated their personal informal learning as highly effective. By 
contrast, managers and human resource developers typically planned formal programs without providing 
structures to support informal learning (Bolt 2009). Four of the ten key recommendations from the Bolt 
(2009:214) study were to “integrate informal learning into training and development programs and work 
practices”, “enhance managers’ understandings of different types of learning and how they can most 
effectively be used in different situations”, “include transfer of learning strategies” and “adopt a hybrid 
approach that meets individual and organizational needs”. In this study participation in informal learning 
included inductions, mentoring, networking, attending meetings, Internet searches, work shadowing, reading 
and online learning (Bolt 2009: 136, 154). 

Online learning environments can support informal learning by facilitating collaboration and reflective 
practices, thus, enabling participants to construct knowledge and transfer learning through, for example; 
journaling, conversing asynchronously and synchronously, sharing audio and video (Zepeda 2012; Zygouris- 
Coe and Swan 2010). Lloyd and Duncan-Howell (2010) predict that in the future professional development 
will occur as self-directed individuals fluidly move in and out of online communities of practice; learning 
from each other just-in-time. So, if there is a destination - are we there yet? 
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4. CONCLUSION 

A brief overview of literature on professional development indicates we are not there yet (Zepeda 2012). 
Online resources have the potential to be used flexibly and widely to support teacher professional 
development (Zygouris-Coe and Swan 2010). An advantage of online professional development is that it can 
facilitate both formal and informal learning which can be accessed just-in-time and sustained over-time. It 
allows teachers to form communities of practice and collaborate with people beyond their face-to-face 
associations in both time and space. A significant point raised in this paper is that elements characteristic of 
effective professional development are similar for face-to-face and online situations (Bolt 2003: Lloyd et al. 
2005). However managers and developers in some large organizations do not plan for informal learning to 
occur (Bolt 2009). Managers and developers face challenges due to rapid changes in technology and 
participants’ ability to use it. Also, it is challenging to design, implement, maintain and assess the 
effectiveness of online professional development programs (Zygouris-Coe and Swan 2010). Even so, the fast 
paced dynamic environment of the 21 st Century demands agile alternatives to traditional methods. As 
workforce dynamics change and the proportion of technologically savvy staff increases there will be greater 
demand for just-in-time flexible learning opportunities. As more teachers use technology to teach their 
students, the need to provide them with online professional development will increase also. This paper 
identifies the need for development of informal online professional development opportunities to support 
knowledge creation and transfer of learning in a range of settings. Such developments would require 
evaluation and dissemination of findings to the broader academic and educational community. 


REFERENCES 

Bolt, S. 2003. Teacher identification of significant action learning experiences leading to professional growth. 
Unpublished Thesis, Curtin University of Technology, 2003. 

Bolt, S. 2007. A framework for improving teaching and learning through action learning. Proceedings of the Fifth 
Education in a Changing Environment Conference. Salford, England, pp. 275-282. Retrieved 12 July 2012 from 
http://www.ece.salford.ac.uk/proceedings/papers/28_07.pdf 

Bolt, S. 2009a. An Investigation of professional learning in dynamic environments. Unpublished Thesis, Curtin 
University of Technology, 2009. 

Bolt, S. 2009b. A contingency approach: Building the capacity of managers to facilitate professional learning in the 
workplace. The International Journal of Knowledge, Culture and Change Management , Vol. 9, No. 5, pp. 65-78. 

Bolt, S. 2010. Synchronising formal and informal learning in the workplace to enhance professional development. In 
Westover, J. 2010. Training the Next Generation of Knowledge Workers : Readings for Effective Secondary Education 
and Workplace Learning Practices. Australia: Common Ground Publishing, pp. 187-201. 

Burns. R. 2002. The Adult Learner at Work (Second Edition). Allen & Unwin: Sydney. 

Caffarella, R.S. 2002. Planning programs for adult learner: A practical guide for educators, trainers and staff 
developers. San Francisco: Jossey-Bass. 

Creswell, J.W. 2008. Educational Research: Planning, conducting and evaluating quantitative and qualitative research. 
Pearson Education Inc: New Jersey. 

Guskey, T.R. 2000. Evaluating Professional Development. Corwin: California. 

Lloyd. M., Cochrane, J. and Beames, S. 2005. Towards a model of effective professional development in ICT for 
teachers. Commissioned report for Queensland Society for Information Technology in Education (QSITE). Retrieved 
12 July 2012 from http://www.qsite.edu.au/publications/position/PD 

Lloyd. M. and Cochrane, J. 2006. Celtic knots: Interweaving the elements of effective teacher professional development 
in ICT. Australian Educational Computing, Vol. 21, No. 2, 16-19. 

Lloyd. M. and Duncan-Howell, J. 2010. Changing the metaphor: The potential of online communities in teacher 
professional development. In Olofsson, D. and Lindberg, J.O. 2010. Online Learning Communities and Teacher 
Professional Development: Methods for Improved Educational Delivery. Information Science Reference: New York, 
pp. 60-76. 

Zepeda, S.J. 2012. Professional Development: What Works (Second Edition). Eye on Education : New York. 

Zygouris-Coe, V.I. and Swan, B. 2010. Challenges of online teacher professional development communities: A state 
wide case study in the United States. In Olofsson, D. and Lindberg, J.O. 2010. Online Learning Communities and 
Teacher Professional Development: Methods for Improved Educational Delivery. Information Science Reference: 
New York. pp. 114-133. 


290 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


DEVELOPMENT OF VISUALIZATION OF LEARNING 
OUTCOMES USING CURRICULUM MAPPING 


Takashi Ikuta and Yasushi Gotoh 

Niigata University 

8050, Ikarashi 2no-cho, Niigata City, Niigata, Japan 


ABSTRACT 

Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS). The objective is 
to have groups of teachers belonging to educational programs discuss whether visualized learning outcomes are 
comprehensible. Discussions based on teachers' subjective judgments showed in general that visualized learning 
outcomes express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level of 
success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting and lessons 
and assessments that correspond to weighting. 
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1. INTRODUCTION 

Amid the increasingly fast-paced development of science and technology and the globalization of economic 
activities, how to assure graduate attributes has become a matter of the utmost importance. As graduate 
attribute frameworks, Japan has “graduate power” and debates about quality assurance in different fields on 
the U.K. QAA model. Our purpose, in terms of such quality assurance, is to make the basic knowledge and 
understanding which students must acquire as their grounding in a relevant discipline; the domain-specific 
skills; the generic skills; and the learning methods and grade assessments needed to obtain such knowledge 
and skills, reference points in forming educational curricula in different fields. At present, trials are underway 
in the fields of management, languages, literature, law, biology, home economics, mechanical engineering 
and mathematical science. 

In parallel to this type of discussion about quality assurance in different fields, it will also be necessary 
for teachers' groups to discuss individual educational programs. As Kawashima (2008) points out, teachers' 
groups involved in educational programs must discuss learning outcomes with the aim of fostering human 
resources. If a teacher is not able to perceive learning outcomes as being related to the actual learning 
outcomes of her own students, it is highly likely that quality assurance in different fields will end up as an 
abstract concept. 

At the same time, there are a number of difficulties in the visualization of learning outcomes. In PDP 
(Personal Development Planning) advocated in the U.S., students themselves perform reflection on the basis 
of the results transcript provided by the university and the learning process record known as PDR ( Personal 
Development Records) compiled by the student. This is supposed to foster students' ability to plan their own 
lifelong development, but it has also been reported that PDP is time-consuming and expensive and hinders a 
proper grasp of learning outcomes (Benesse). What is needed is a system which enables students to visualize 
learning outcomes for attainment targets. 

Niigata University has started to develop the Niigata University Bachelor Assessment System (NBAS) on 
the basis of these considerations. This system sets lower attainment targets in four educational target 
domains, namely: knowledge and understanding, domain-specific skills, generic skills and attitude, and 
visualizes these by displaying on a radar chart students' level of attainment for the corresponding learning 
outcomes. By visualizing learning outcomes as a radar chart display, it becomes possible to discuss students' 
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quality assurance from the point of view of generic skills and attitude, as well as from the point of view of 
knowledge and understanding and domain-specific skills. 

Ikuta and Gotoh (2011) proposed PDCA by teachers' groups on the basis of the relative importance of 
educational target domains in educational programs. Teachers' groups have a bird's eye view of the 
curriculum as a whole, but if quality assurance of graduate attributes is not to end up as an abstract concept 
without any substance, they need to examine concrete learning outcomes visualized for each student. More 
specifically, what they must do is verify whether visualized learning outcomes are comprehensible, i.e. 
whether the radar charts displayed by the system reflect students' attributes and skills. 


2. OBJECTIVE 

The objective is to have groups of teachers belonging to educational programs discuss whether visualized 
learning outcomes are comprehensible. 


3. METHOD 

Teaching staff from the life sciences, forest environment studies, and agricultural engineering programs and 
teachers from the Institute of Education and Student Affairs took part in the study. The study started in 
October 2010 and is scheduled to end in June 2012, i.e., it is still ongoing. 

The visualization method for learning outcomes is the one shown by Ikuta and Gotoh (2011). First, the 
educational targets domain was split into domain-specific academic knowledge, domain-specific skills, 
generic skills and attitude, and attainment targets were set at the lower level of this educational target domain. 
Next, a contribution ratio was assigned to the attainment targets in each subject forming an educational 
program. For example, in subject A, the attainment target in knowledge and understanding was 50%, 
domain-specific skills 30%, generic skills 10% and attitude 10%. In the educational target domain, several 
attainment targets were set. For example, where there were contributions to several attainment targets in 
knowledge and understanding, a further 50% was allocated to each attainment target. A curriculum map 
drawn up in this way is called a “weighted curriculum map.” In providing such weighting, lessons and 
assessments must be carried out by a corresponding method. 

Feaming outcome was the total score obtained by multiplying a student's grade assessment by this 
contribution ratio and the number of credits. To give an example, if a student scored 80 points in subject A, 
his score for knowledge and understanding was obtained by multiplying 80 (points) x 0.5 (50%) x 2 (number 
of credits) to achieve a score of 80. This score was totalled for each attainment target to obtain the learning 
outcome. Weighting was performed from October 2010 to March 2011. 

To find out whether intuitive comprehension of learning outcomes visualized on the basis of actual 
students' results was possible, students' performance data up to March 2012 (data at the end of the third year 
of the life sciences program, and data at the end of the fourth year of the forest environments studies and 
agricultural engineering programs) were used in order to draw up a radar chart. At the same time, materials 
were also prepared for understanding a student's relative position on each attainment target. 

Using these data, teachers belonging to the programs were interviewed and the results analyzed. The 
analysis transformed speech data into textual data, split these data for each segment and then categorized 
them. Although three educational programs were examined in this study, Niigata University offers 42 
programs and in future the data must be supplemented and verified. For this reason MAXQDA10 was used 
as the analysis software. 


4. RESULTS 

4.1 Visualized Learning Outcomes are Comprehensible 

In general, visualized learning outcomes clearly express students' particular characteristics and are 
comprehensible. In forest environment studies, the nature of the curriculum means that the subjects studied 
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by students are more or less identical and in many cases the shape of the radar chart display was similar. The 
difference between learning outcomes for students with good grades and students with poor grades was also 
immediately obvious from the size of the radar chart. (Figure. 1) 


Knowledge end understanding. Domain-specific skills. 





2008 ( 1 ) 

2008 ( 2 ) 

2009 ( 1 ) 

2009 ( 2 ) 

2010(1) 
2010(2) 


Figure 1 . An example of visualized learning outcomes 

By looking at the shape of the radar chart for each student, it is possible to understand which specialized 
domain subject students were learning and when. Students with an extremely low GPA have a distinctive 
radar chart display and since they can grasp visually which attainment targets have not been reached teachers 
are likely to find such radar charts useful when advising students. 

In life sciences, there are four specialized domains: housing studies, clothing studies, dietary habits and 
family resource management and since the subjects taken by students in each of these domains are virtually 
identical, typical patterns were generally confirmed for all four. 

It was proposed that using visualized learning outcomes in this way to carry out peer reviews among 
students in the same domain might be useful in setting targets for the next stage. 


4.2 Need for Pointers for Self-Assessment of Learning Outcomes 

In order to enable students to self-assess their own learning outcomes, some teachers thought that they should 
be provided with some kind of indicator or pointer. The idea is that for example, if one role model was drawn 
up for forest environment studies and four role models for life sciences in the domains of housing studies, 
clothing studies, dietary habits and family resource management, respectively, and if students could compare 
themselves to these role models, self-assessment would become easier. 

There are a number of methods concerning which pointers to use. Suggestions included: using actual data 
from excellent students and virtual data; showing the minimum line; comparing one's own performance with 
the standard performance by using the average value and the mode in subjects taken by that student; or 
combining these methods, i.e. comparing one’s own position to both the excellent line and the minimum line. 


4.3 Need for Continuous Examination of Curriculum Map Weighting is 
Needed 


When the relative position of students for each attainment target was discussed, attention was drawn to the 
fact that some attainment targets did not seem right. Some students who had a feeling that in terms of 
attainment targets they should be placed higher up, found that they were not particularly highly placed. 
When the weighted curriculum map was checked, it was found that only a small number of subjects had been 
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weighted for that attainment target and students who scored poorly had done so because they had not taken a 
weighted subject. 

On this basis, it seemed clear that further debate was needed about whether attainment targets were too 
fragmented or whether there were too few weighted subjects. 

4.4 Need for Discussion about Whether Lesson Contents and Assessments 
Correspond to Weighting 

Where teachers' groups in life sciences programs weighted curriculum maps by giving, for example, 50% to 
knowledge and understanding and 30% to domain-specific skills, they shared a recognition that lessons and 
assessments must also be carried out by corresponding methods. As a result, compared to pre -weighting 
(around March 2011), at present (March 2012), teachers are more aware of the relationship between 
attainment targets and lessons and assessments and revision is possible. In addition, attention was also drawn 
to the fact that teachers had greater awareness of the connection between other subjects making up the 
educational program and their own subject. 


5. DISCUSSION 

Discussions based on teachers' subjective judgments showed in general that visualized learning outcomes 
express students' natural abilities and, unlike simple GPA and grade data, are conducive to grasping the level 
of success in reaching attainment targets. In future, it will be necessary to examine amendments to weighting 
and lessons and assessments that correspond to weighting. To identify the reliability of visualized learning 
outcomes, some kind of direct assessment method such as learning portfolio or performance assessment will 
be need. 

Ikuta and Gotoh (201 1) proposed that “If individual teachers revise the syllabus, extract attainment targets 
and consider assessment methods and grade allocations in the educational program as a whole, they will be 
able to discuss educational programs for fostering human resources in general not only from the point of 
view of domain-specific knowledge, but also from the point of view of generic skills and attitude. Such 
discussions will have wide-ranging potential application in assuring quality in higher education.” 

Among teachers belonging to educational programs, there are disagreements even about individual 
generic skills and discussions may finish in abstract definitions. We want individual teachers to say, “What 
do these generic skills mean in concrete terms for students in my class? What lessons should I design to 
foster such skills? How should I assess them?” For this reason, as in the present study, it is essential to 
continue to examine how learning outcomes are being visualized for real, individual students and to provide 
weighting to the whole curriculum by taking a panoramic view of it. 
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ABSTRACT 

The virtual reality is being more and more used in the education, enabling the student to find out, to explore and to build 
his own knowledge. This paper presents an Educational Software for presence or distance education, for subjects of 
Formal Language, where the student can manipulate virtually the target that must be explored, analyzed and studied. 
With a simple interface, with easy comprehension and using, this paper presents an educational software where the 
student manipulates the objects that will be studied in 3D, getting easier the study of concepts and theories about 
Automatons. Regular Expressions and Minimization of Automatons interacting directly with the object in 3D. To the 
design of the automatons, the software uses tolls in 3D, as the Blender and the VRML (Virtual Reality Modeling 
Language) and to the publishing of a page on the internet it is integrated the Program Language PHP (Hypertext Pre 
Processor). The results reached with the use of the developed software show the attributes that make the ideal Virtual 
Reality for situations of research and learning taking the discipline as a reference of the classroom to the computer labs 
and making more interesting to the student, making the learning easy. 


KEYWORDS 
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1. INTRODUCTION 

The constant evolution of the technology is taking the education to new ways, much more attractive to the 
students, making possible the use of new tools, taking to an evolution on the teaching process. The Virtual 
Reality takes an important place in this evolution. 

Several are the definitions about the virtual reality, but in general, they refer to the an immersive and 
interactive experience based on graphic images in 3D generated in real time by computer, in other words, it is 
a simulation generated by a computer, about a real or just an imaginary world. Another definition, more 
specific is: virtual reality is a computer interface that permits the user to interact in real time, in a 
tridimensional space generated by a computer, using their feelings, through special devices (Kimer, 2012). 
The user ca notice the virtual world, through a window built by the monitor screen or by projection screen or 
it can be inserted in the real world through a helmet (HMD) or multi projections rooms (caves) and 
interaction devices (Kimer, 2012). 

The education can be seen as a discovery, exploration and observation process, besides the eternal 
construction of the knowledge. With this, the specific characteristics of the virtual reality can transform it on 
a mighty tool in service for everybody who seeks for an education evolution. Many things that until short 
time ago were dreams, nowadays, with the current technological advances became reality. 

With the virtual reality acting in the education we can discover, explore and build knowledge about places 
and situations that we could never explore. The great potential of the virtual reality is exactly on these 
possibilities, not only through classes or physical objects, but also through the virtual manipulation of the 
target to be explored, analyzed and studied. 

We can understand as Formal Language Theory and of the automatons, the study of the math models that 
enable the specifications and the language recognizing (in the wide sense of the word), their classifications, 
structures, priorities, characteristics and inter-relationships (Furtado, 2012). The importance of this Theory in 
Computer Science is double: it supports as the other theoretical aspects of the Computer Science, as it 
justifies several computer applications such as language processing, pattern recognizing, modeling systems 
(Furtado, 2012). 
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The proposed educational software has as a purpose to help students and teachers to have a more 
complete and interesting view about the automatons and about the changes occurred on their minimization. It 
also has as a purpose to make the subject more interesting to the students, making the learning simpler. We 
intend to make the student more participative, making the learning interactive. Trough the software 
developed it’s also possible to evaluate the learning of the students on the distance modality, making easier 
the teacher’s work who receives in his registered electronic address the result of the exercises done by the 
students on the 3D software. 


2. VIRTUAL REALITY 

Virtual reality, augmented reality and their variations represent computer interface techniques that take into 
account the tridimensional space. In this space, the user acts in a multi sensorial way, exploring aspects of 
this space through the viewing, hearing and tact. According to the available technology it’s also possible to 
explore the smell and the taste. Body perceptions, like cold, heat and pressure, are included in the tact, 
through the skin (Kimer, 2011). 

Virtual reality is characterized by three basic ideas: (Pinho, 2004) 

• Immersion: the user has the real sensation of being inside the virtual world of the computer. Devices 
that make this sensation: digital helmets and digital cave. 

• Interaction: the user manipulates virtual objects. Devices that make this sensation: digital gloves. 

• Involvement: exploring of a virtual environment, it’s as if the user took part of the virtual world and 
he can interfere directly in result of the application, the user can navigate on the virtual environment 
in a passive or active way. 

2.1 Virtual Reality in the Education 

The technological revolution has been permitting the use of new approaches in the teaching-learning process. 
One of the conductive technologies to the building of innovative tools for the education is the Virtual reality, 
which offers tridimensional computer environments with advanced forms of interaction that can provide 
more motivation to the learning process. 

A very short time ago, we could consider that the great potential of VR use was in small groups placed in 
large urban centers and in teaching and researching institutions. However, the integration VR-VRML 
democratized its access, expanding more and more its potential and using fields (BARILLI et al, 2012). 

With the help of resources of some modeling and animation programs as the Blender 3D, for example, the 
VR use can help students in the comprehension and assimilation of concepts, coming up as a valid alternative 
to get good results. Other benefits are observed with the use of Virtual Reality in the education. 

According to Clark (2006) the Virtual Reality can be used to make the learning more interesting and fun 
with the purpose of improving the motivation and attention, decreasing costs when using the objective and 
the real environment no matter how expensive the simulation is. It also makes possible that situations that 
were impossible to explored in the real world can be done, for example: exploring a planet like Mars, 
traveling inside the human body, doing submarines explorations or inside caves, visiting very small places to 
be seen (molecules) or very expensive or very far away, or yet because this place is in the past (historical 
places). 


3. FORMAL LANGUAGES 

The Formal Languages were developed in 1950 with a purpose of developing theories related to the natural 
languages. (Menezes, 2005) But soon it was checked that the formal languages were ideal to the languages 
study in the Computer Science area. In the languages study two kinds of problems to be solved and treated 
were found, the syntactic and the semantic. The syntax treats about the grammatical checking of the 
programs, the free properties of the language, as the meaning of a program. 

The study of the Regular Languages is done starting from three formalisms (Menezes, 2012): 
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• Finite automaton: it’s the recognizer formalism, being a set of states of a finite system; 

• Regular expression: it’s the denotational formalism, it defines how to build the words of the 
language, it’s a basic set of language operation; 

• Regular grammar: it’s the generator formalism that corresponds to the production rules. 

A finite Automaton is a system of sequential finite states that represent a model of Computer Science, 
widely used in the Formal and Compiler Languages, being used to studies of Formal Languages theory in 
Computer Science. A Finite Automaton can be deterministic, non-deterministic or with empty movements 
(Menezes, 2012). 


4. SOFTWARE DEVELOPMENT METHODOLOGY 

The educational software developed is based on the Virtual Reality oriented to the Formal Language subject, 
having as a target the study of Automatons Minimization. To the mounting of this software a Blender 
modeler was used, enabling the building of images in 3D of the automatons. 

The Blender modeler permits to explore its files to the VRML 2.0 language that is a language used to the 
descriptions of virtual worlds enabling this way that corrections on the images be done using the command 
lines. 

It’s also necessary that the Plug-in Cortona be installed on the computer where the software will be used. 
The Plug-in opens the navigator with controls that permit that the user to visualize the images by different 
angles, enabling thus, the interaction of the student with the objects to be studied. 

The Blender modeler used to the building of the Automatons images don't permit that letters and numbers 
be exported, so it’s necessary that the letters and numbers, as for example: ql, q2, a, b, and 0 be placed on the 
images through the VRML Language. 

The software is available through a webpage on the internet, where the students have access to the initial 
page. 

Through the initial page the users will be able to access the Menu of the page that shows the study option, 
it presents to the student the possibility of downloading the Plug-in Cortona and it also leaves available some 
links for reading. 

The software will be available through pages built using the PHP program language. The student will be 
able to choose to answer questions about regular expressions before performing the Automatons’ 
Minimization. 

The automatons’ minimization is done on a screen where the user visualizes the Automatons' design in 
3D and he must insert some state that is necessary ant to use the table and the lists to find the equivalent 
states that can be removed. When the user finds the equivalent states, he will be able to visualize the 
minimized Automaton in 3D. If the student doesn't get the right equivalent states, the page will advertize him 
about the error and the student will be able to repeat the exercise. 

The page presents exercises with multiple choices, where the student is going to answer the questions and 
check if the chosen answers were correct, this way the student will be able to test his theoretical knowledge. 
The page advertises the student when the options done in the questions of multiple choices are the correct 
ones and the student will have the option of repeating these wrong questions. 

The page presents a Material for Reading that is destined to explain doubts about Regular Expressions 
and Automatons’ Minimizations. This page can also be used by educators in their presence or virtual classes. 
The page also presents a help material that explains how to use the pages with the Minimization exercises. 
This help is available through the Menu of Minimization. 

If the material for reading doesn't solve all the doubts, some links that were used in the work are also 
available in a web page. The answers of the questions of Minimization are also available in the page of the 
subject, and they can be accessed through the Help page. This page is to help teachers and students, if some 
problems about solving these exercises happen. 

Using the system above with a package of distribution apache with PHP provider that owns the 
phpMyAdmin toll, a bank of data was created to store the login and the students’ password to do the 
evaluation. The student's login must be the email that he desires to receive his evaluation, and the teacher 
will receive in an email that is already defined in the program. To facilitate the use, the bank will be exported 
and it must be imported by the teacher who wants to use the system. 
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Then it has done a checking to check if the answers are correct. When all the exercises are done the 
student will send the results and the grade automatically to the email address registered in the beginning like 
a login and to the teacher’s email, so this way the evaluation is done automatically. 


5. CONCLUSION 

The educational software helped to take the Formal Language Subject from classrooms to the computer labs, 
making the teaching/learning process more interesting and pleasant to the students, facilitating the teacher’s 
work during the evaluation performance too. 

The Virtual Reality presents an opportunity of learning with a real situation, but artificially created, 
facilitating the visualization and the interaction sensation with the study focus. When we can't have the real 
experiences, the Virtual Reality is irreplaceable. The simulation in the VR also permits us to be in hard and 
dangerous situations, which aren't usually accessible in the real world. Furthermore, the VR permits to take 
to the students complex themes of hard learning and sometimes impossible to show. 

The software presented was used during a year in the Formal Language Subject, at the X University, 
where it was found a better appropriation of the concepts and a bigger facility in the activities performance 
with automatons' minimization, resulting thus in an improvement in the students’ evaluation who used this 
software with VR. 
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ABSTRACT 

The paper shows the importance of Storybird, an online platform, for developing writing and storytelling among young 
learners of a foreign language. 
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1. INTRODUCTION 

Story telling is one of the most ancient forms of human expression. In an increasingly complex and 
media-saturated world there are a vast range of digital tools that allow students to express through digital 
narratives. Digital storytelling emerged over the last few years as a powerful teaching and learning tool that 
engages both teachers and their students, (Robin, 2008). Teachers must invest more time/effort in getting 
both content producers and consumers to develop their storytelling and narrative skills. 

There are several definitions for ‘Digital Storytelling' however we identify with the one from the Digital 
Storytelling Association that defends that “Digital Storytelling is the modern expression of the ancient art of 
storytelling. Throughout history, storytelling has been used to share knowledge, wisdom, and values. Stories 
have taken many different forms. Stories have been adapted to each successive medium that has emerged, 
from the circle of the campfire to the silver screen, and now the computer screen." (Leslie Rule,2010) 

Digital stories usually contain computer based images, text, recorded audio narration, video clips and 
music. They vary in length but in education the most used last between two and ten minutes. Topics range 
from a wide area, covering almost everything thinkable. 


2. STORYBIRD 

Storybird is an extremely engaging collaborative storywriting website that embodies three ideas - creating, 
reading, and sharing. It is also a collaborative storytelling tool that allows students to focus more on the 
content of their writing rather than drawing pictures. Students are provided with the pictures - free collections 
of art. They just have to add the words to write stories. Once the art is chosen, students are able to build their 
story by dragging and dropping pictures and creating/writing a story to match the pictures chosen. Stories can 
enclose a variety of genders - poetry, mysteries, tales, among others. 

For teachers it is very easy to use because Storybird requires minimal teacher preparation and allows 
them to easily create individual user accounts for students. With that, teachers can view all story books that 
students are making. Storybird also has online safety for young students built into it. Storybird can be used 
collaboratively with, either with another student in class or school, or with students from different schools in 
the region or even from another country. 

Storybird can also be used as a way to collaborate between parent and child. Several experiments were 
made in the USA in order to bridge home and school by having a child and their parents writing pages in a 
book. Afterwards schools have a book share day where parents come to school and read the story with their 
children to the rest of the class. 
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Storybird is also excellent for many writing assignments (feature built in the platform) helping even the 
most hesitant writers to bring out of their shell as this can be considered an engaging activity. 


3. APPLICATION IN EDUCATIONAL CONTEXT 

Storybird was introduced and demonstrated in a Language Teacher Master class in the Polythecnic of Castelo 
Branco, Portugal. It arose enthusiasm among the student teachers and so they decided to design activities to 
be used in schools with their own students. As there were teachers from different levels of teaching, activities 
were designed for kindergarten students up to secondary students. I'm going to describe the implementation 
and outcomes of some of those activities below: 

1. The first one was implemented in two Primary Schools and intended to pair up students from two 
classes in different schools in the region. They had to write a story together. This was very important because 
each student brought his/her own style to the story while working together. First they chose a theme. Then 
they were given a vast array of pictures that could accompany that theme. They had to choose the pictures for 
the story and then drag and drop each picture onto the pages. Afterwards they discussed orally about the story 
and then wrote it. The fact that the work could stay unpublished until it was finished allowed it to be worked 
on and improved over several class periods. Each class worked on the stories at their own pace and rhythm 
and once the stories were finished, students read/tell the story via Skype to the other class. Afterwards the 
story was published online and printed. 

Writing and reading for an audience encouraged and developed literacy skills. Storybird also helped the 
reluctant and shy students to write on their own. The use of artwork allowed students to develop deep 
reflection and higher-order thinking. 

2. Progressive story project 

This activity was developed in a language class in a Basic School. The class was divided into groups. One 
group of students started the story and then the next day, the rest of the students read what the first group 
wrote, then the second group added to it and so on until the end of the story. All the students participated 
actively. Students loved seeing how the story that they started went on and ended. 

In this case it was not only a great writing opportunity, but a great reading one as well. Creativity was 
high. All the students wanted to surprise the group that was coming afterwards. 

3. In a Kindergarten, the experiment was done orally, using the art available. Children were asked to tell 
the story. When they did not agree, they negotiated another development for it. With these wordless picture 
books Storybird promoted early writing and early story telling creatively. 

With these young students it was also used to develop sequencing skills and understanding the parts of 
the story (reading/writing skills). 

There were many other activities developed by the students. These were only some examples. 


4. CONCLUSION 

Engaging in activities which students see as having a concrete and practical outcome, such as writing 
electronic books, allows the students to be creators of something unique, of which they have ownership. They 
become comfortable with the act of creation: turning nothing into something. 

It also facilitates the transition from teacher-centred, class-based learning to one in which the pupil begins 
to acquire individual responsibility. 

Because the work is carried out and published online, new, previously inconceivable possibilities of 
peer- and self-assessment also emerge in which students start learning via interaction with others. 

The art pictures were so good that the students in the activities described above, were inspired to write 
and read the colleagues’ stories. The more they read the more they wanted to write in order to read more 
stories to the classmates. This was very important because students were able to understand concepts and 
ideas better. The sharing of their stories made them comment, negotiate and discuss, giving them the tools to 
live in society. 

Storybird promoted imagination, literacy, and self-confidence. 
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ABSTRACT 

There are some mathematical learning models of collaborative learning, with which we can learn how students obtain 
knowledge and we expect to design effective education. We put together those models and classify into three categories; 
model by differential equations, so-called Ising spin and a stochastic process equation. Some of the models do not contain 
interactions among students and we discussed the possibility of improvement of the models for collaborative learning. 
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1. INTRODUCTION 

In recent years, the theoretical study of teaching-learning processes has attracted the attention of an 
increasing number of scientists. Early studies on those processes have been conducted by psychologists and 
sociologists (Piaget, 1929; Vygotsky, 1978). Although the topic is so difficult and complex that there are still 
many open questions, the study of cognitive processes has developed into an active area of multidisciplinary 
investigation as a number of physicists have got interested in research areas such as economy, social science, 
biology and so on. Since Hake reported that the performance of the students can be enhanced using a 
teaching approach involving collaborative group work, in contrast to the traditional non-interactive lectures 
(Hake, 1998), the processes of learning and understanding physics and mathematics have become the focus 
of cognitive research. In order to precede the research, mathematical model of teaching-learning process have 
been proposed and studied. 

The aim to study teaching-learning process by the use of mathematical model is to investigate the 
influence of the structure of the group works on the achievement of the students and to design effective 
education. In this reflection paper, we put together mathematical models of learning process and summarize 
some aspects of those models. 


2. MATHEMATICAL LEARNING MODEL 

There are some mathematical models of teaching-learning process and we classify these models into three 
categories in the following subsections. 

2.1 Differential Equation Modeling 

Pritchard et al proposed models of teaching-learning process by the use of differential equation (Pritchard, 
2008). The models are based on various theories of learning: tabula rasa, constructivist, and tutoring. They 
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predict the improvement of the post-test as a function of the pre-test score depending on the type of 
instruction. One of the models is the connectedness model combined with tabula rasa and constructivist. The 
connectedness parameter measures the degree to which the rate of learning is proportional to prior 
knowledge; dU T / dt = —j3aK T U T — (l — j3)a'U T . U T and K T are fraction of unknown and known in a 

test domain T respectively and they are dependent of the amount of instruction t . CX and a! are control 
parameters. The first term of the right hand side (rhs) is motivated by the constructivist view that students 
learn new knowledge by constructing an association between it and some prior knowledge. The second term 
of rhs is motivated by the tabula rasa theory of learning. 

Although the model itself is quite simple and an exact solution can be obtained analytically, it fits existing 
data by sharply determining a parameter. However, these models do not describe a collaborative learning 
among students. 


2.2 Ising Spin Modeling 


Bordogna and Albano proposed a mathematical model of teaching-learning process in a classroom by the use 
of a constructive approach (Bordogna and Albano, 2001). In the model, interactions between students and 
teachers are described with a set of equations similar to those that describe magnetism in materials. They 
used so-called generalized Ising spin <7j that is the knowledge of the student j and is defined as a dynamic 


variable such as — 1 < <j . <1. Students play the role of spins and their knowledge of a subject is similar to 

the orientation of the spin. A teacher behaves like an external magnetic field trying to align the student's 
knowledge to the right direction. The model is described as follows. The cognitive impact of the teacher on 

the student j is Cl J S = Pj T (\ — CAOy), where <J T and P T are the knowledge of the teacher and his/her 


ability to persuade the student j, respectively. Within groups of N students, the cognitive impact of the 
student-student interaction is given by Clf S = ^ J [ R(l — CTjCT:) — S.-(l + <T,C,)]sign(cr, / <J T ) , 

J i=[,& j * ’ 1 'J N ’ 1 1 1 

where /■?. and Sjj are mutual persuasiveness and support of student i to j respectively. The knowledge ex- 


changes according to the total cognitive impact stochastically by the use of Metropolis algorithm. 

The simulation results of the model is consistent with well-established empirical results, such as the 
higher achievements reached by working in collaborative groups and the influence of the structure of the 
group on the achievements of the individuals. Yasutake et al introduced network structures formed by 
students into the model and investigated some effects of the network structures (Yasutake, 2011). 


2.3 Stochastic Process Modeling 

Nitta developed a phenomenological theory of peer instruction (Nitta, 2010). His model describes rather a 
short-time process of learning, in contrast to the models presented in the above subsections that describe 
long-term learning gain. He modeled the transition of the number of students answering correctly for 
multiple-choice questions (MCQ) after discussions among students by the use of a master equation. By 
denoting the normalized number of students choosing the answer a for the MCQ q before discussion and 
after discussion as p x ( Q. ; 3) and p n (Cf,3) , respectively, p 1 ( Cf, 3) is described as 

Pi (3 c ) = P\ (Cf, C) ~ 'Yj di /Cj T dc ( q)p { + Q) A (qd), where c and d represent the correct 

answer and distractors, respectively and 7^(Q) is the transition matrix that represents the normalized 

transition rate of students from answer b to a on MCQ q. Then the master equation was simplified 
analytically and he showed that the number of correct answers after peer discussion is approximately given 
by a simple function of the number of correct answers before discussion as p 1 = p { + p { (1 — p } ) . 

The theoretical curve agrees with data obtained from lectures implementing the peer instruction. In 
addition, it was shown that a differential equation derived from the model corresponds to the simple 
connected model of Pritchard et al. However details of the peer instruction is not included in the model. 
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3. REFLECTIONS 


Although Ising spin model describes the mutual interaction among students and teachers, differential 
equation model and stochastic process model do not include elements of collaborative learning. In this 
section, we would like to suggest the possibility to extend those models in order to take into account 
collaboration among students. 

There is a simple extension in the differential equation model described in section 2.1. The unknown 
fraction of a student i in a test domain Uj should decrease with the help of other wiser students and could 


rather increase because of confusion caused by discussion with less wise students. One possibility to 
implement the collaboration is to add a term —oc"/ , (K — /C ), where K- is a known fraction of a 


i*> 


student i and the subscript T for test domain is omitted here. 

It is not straightforward to improve the stochastic process model described in section 2.3. Collaboration 
should emerge as a result in elements of the transition matrix 7^(Q). Therefore we have to model each 


student’s behavior intended to choose an answer for MCQ before and after discussion and make up the result 
statistically to construct the transition matrix. This extension requires further consideration. 


4. SUMMARY 

We briefly summarized three kinds of models for teaching-learning process; differential equation modeling, 
Ising spin modeling and stochastic process modeling. Mathematical modeling approach to design effective 
teaching-learning environments has just started and the study hovers at a level of applying models to each 
individual case. Further investigation and more general model would be expected. 
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ABSTRACT 

Games are currently being used by an increasing number of people, of different ages and in different contexts. Usually, 
they provide fun ways of interaction, collaboration, and competition among players. These aspects may be very important 
in an educational context, since many research studies confirm that games have many positive effects on students when 
used as learning activities. Therefore, we present in this paper a Collaborative Quiz game concept, which promotes 
collaboration between students and motivates them for learning. This game will be created using a graphical development 
environment currently under development, denominated Epik. 


KEYWORDS 

Collaboration, Interaction, Educational Games 


1. INTRODUCTION 

Educational games, compared to other types of learning materials, can transmit new concepts to players in a 
much funnier, interactive and dynamic way. Their use in education can contribute to an increase of students’ 
motivation and skills acquisition (Sindre 2009). When accompanied by means of collaboration they also 
allow the development of skills like communication, cooperation and coordination, all relevant in teamwork 
and other daily tasks (Zea 2009). 

Despite these advantages, games are not yet widely used in education, mainly due to difficulties involved 
in their development and distribution (Torrente 2009). However, in recent studies (Torrente 2009 and 
Moreno-Ger 2008), ways to solve some of these problems were already identified, involving the creation of 
tools that enable the development of educational games by teachers, and the use of Learning Management 
Systems (LMS) as a means of delivering them. 

Therefore, we decided to create a graphical development environment for collaborative and competitive 
games development, denoted Epik (Edutainment by Playing and Interacting with Knowledge) (Sampaio 
2012). This environment will support Moodle (Modular Object-Oriented Dynamic Learning Environment) 
integration, allowing teachers to deliver the games as Moodle activities and to reuse learning materials 
produced in a Moodle course. Epik will allow the development of collaborative quiz games, mainly because 
quizzes are one of the most used learning activities. However, they usually don't allow collaboration and 
competition among students, which will be the main difference between the usual quizzes and Epik games. 

In the following section we'll characterize a collaborative quiz game to be developed with Epik and then 
we’ll present some conclusions. 


2. COLLABORATIVE QUIZ GAME 

We will use Epik to create a collaborative quiz game with several multiple choice questions related to a 
certain topic. These questions will be associated with different scores and some types of help will be 
available for players, a bit like in traditional quiz games. The helps and scores shall serve as a way to foster 
collaboration and competition among the players. 

The game shall be played by a group of students who may be in the same physical location or 
geographically distributed. This game will be organized in different scenarios, each composed of an equal 
number of questions for each group member. To progress in the game it will be necessary that all group 


308 



IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2012) 


members respond correctly to the questions from the current scenario (coordination). If any member of the 
group is in doubt about a question, he or she could ask for help. Helps may involve interaction with other 
group members (communication and cooperation) and may be used a limited number of times throughout the 
game. The types of help available will be: (i) learning materials consultation - such as images, audio, 
videos, or files that contain the information needed for quick reference; (ii) request hints from other 
players - there will be a list of hints associated to each question, from which a team member can choose the 
most suitable to help his colleague; and (iii) incorrect answers removal by a player - a team member that 
already answered correctly to that question can help is colleague by removing some of the incorrect answers 
(also known as 50/50). 

The scores associated to each question may vary depending on different factors such as time to answer, 
number of failed attempts and helps used. When a player correctly answers a question, he or she will 
immediately receive the currently associated score. If the answer is incorrect, the question score will be 
reduced in a percentage of its total value. The first player to answer all questions in a scenario correctly will 
receive a bonus with a specified value, which will be a very important aspect to encourage competition 
among the team players. Although, players must also collaborate with each other to progress in the game by 
using helps. By helping a colleague, the player receives a percentage of that question score, while his or her 
colleague receives the question score without suffering any penalties. However, what will determine if the 
help was successful will be the time the player takes to answer the question correctly after receiving help. If 
this time is greater than the given time to answer, the question score will be reduced by half and the player 
that helped will get nothing. 

When the game ends, the team will be ranked based on its final score and other teams scores. This 
ranking is controlled by Epik and will act as a way to encourage competition between different teams. All 
these information elements will contribute as a form of awareness, with regard to the state and actions 
performed by the players. 

Throughout the game it will always be presented to each player the team score (sum of each player total 
score), all of the other players’ scores and their respective avatars. For each player, the scores will be 
displayed and listed by the following categories: correct answers, incorrect answers, collaboration and total. 

Moreover, all the information about team actions (such as, the quickest player to respond on each 
scenario, the number of attempts and helps each one required, the individual player score and the team score) 
will be logged in Epik and sent to Moodle. This information can help teachers with the students’ evaluation. 

Figure 1 shows the example of a game scenario flow, which is related with programming teaching and 
has as main objective the demonstration of the abstract data type List behavior. This game may be an activity 
evaluated in a lab class or a homework activity, for example. In all the scenarios, the players’ scores and 
theirs avatars are displayed on the right side and for each of them their information is highlighted on their 
own screen. At game start the scenario on Figure la is displayed for each player, where some concepts and a 
video are presented to them, in order to explain the List concepts. By clicking on the button at the bottom 
right of this scenario, each player goes into the second scenario (Figure lb), where each one of them needs to 
answer a set of questions (the same for all players). If a player asks for a hint about the upper right question, 
the system selects another player to help him (Figures lb and lc - Anakin asks for a hint and Irina helps him). 



(a) Presenting a concept to players. 


(b) Anakin asks for a hint on the upper right question. 
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(c) Irina chooses and sends a hint to Anakin. (d) The hint and the timer are displayed to Anakin. 


Figure 1. Scenarios' flow of a collaborative quiz game. 

After selecting and sending the hint, it will appear on his colleague screen, beneath the area with the 
question, with the timer next to it (Figure Id). 

All mentioned aspects such as the questions, the topic, the scores, the times to answer, the rewards, the 
penalties and the actions to log, could be defined by the teacher while developing the game with Epik. 
Moreover, the development of these games will be a simple task for the teacher, as their integration in 
Moodle. 


3. CONCLUSION 

With collaborative quiz games we can develop and improve students skills related to the: (i) learning topic, 
since they must answer related topic questions, using the information provided in the game, or possibly the 
help provided by others. Besides that they can also help other students by choosing adequate hints or by 
removing incorrect answers. In both cases the students can acquire and improve their knowledge about the 
game topic; (ii) teamwork, in order to finish the game, students must collaborate, which improves their 
teamwork capacity. 

The competition component introduced by the game scores also motivates the collaboration between 
students and improves their motivation to learn. In addition, the teams’ ranks stored by Epik also motivate 
students to play again and to get better results. 
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ABSTRACT 

Post-secondary students are increasingly receiving instruction by distance learning. This mode of learning can result in 
the students learning in isolation, with bad results in learning outcomes. The same type of isolation can occur for part- 
time students and those who are working while taking classes. In such circumstances, we believe that it would be 
beneficial for instructors to facilitate the formation of small learning communities by distance learning mechanisms. One 
mechanism that can be exploited for this purpose is social networking sites, given their popularity among the student 
population. We are currently developing innovative approaches to building learning communities for distance learning 
students using social networks. As a preliminary step in this research, we have conducted a survey of our target student 
population. In this reflection paper, we present the results of our survey and our reflections on the how the results will 
guide our future work. 


KEYWORDS 
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1. INTRODUCTION 

In order to support students in distance learning as well as part-time students, we are committed to building a 
system for developing learning communities among such students. Research has shown that a shared learning 
experience is vital for student success [1, 2, 3], Our intuition is that social networking sites can be leveraged 
for such purposes due to their widespread acceptance among the target students [4, 5, 6], In order to affirm 
our intuition, we performed a survey of the target students. The results of the survey and our comments are 
presented in the following section. 


2. STUDENT SURVEY AND COMMENTS 

In order to justify our research in using for distance learning with social networking , a survey of the target 
audience - students in a traditional Masters degree program in Computer Science was carried out. The survey 
was meant to gauge their interest in using social networking as part of their study routine, as well as 
preferences for such use. 15 students replied to the survey. The comments on the results follow the survey 
results shown in Table 1. 


Table 1. Survey of Students 


| 1 . In general, which type of class organization would you prefer? | 

Traditional, face-to-face. 80% 

Distance learning. 7% 

No Preference. 13% 

| 2. Do you use social networking services (e.g. Facebook, Twitter, Google+, Linkedln, etc.)? | 

Yes, often. 67% 

Yes, seldom. 33% 

No, never. 0% 

| 3. In preparing for exams, understanding homework, etc. what is your predominant study mode? | 

I study by myself. 

I study in a group with other students 


87% 

from the class. 13% 
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1 4. If you answered "I study by myself" to the previous question, would you study with other students in a group if there was a convenient 1 

| way to do so online? 


1 

[ Yes. 69% 

No. 31% 


| 5. If you answered "I study in a group with other students from the class" to question 3., estimate the typical size of the study group. j 

1 other student. 25% 

2 other students. 50% 

3 or more students. 25% 

1 6. If you answered "I study in a group with other students from the class" to question 3., do you also typically interact with members of 1 

| the study group using a social networking service? 


1 

f Yes. 50% 

No. 25% 

Not sure. 25% 

1 7. Do you currently use social networking services as part of your study routine for classes (e.g. exchanging information about a class | 

| with another student in the class via Facebook)? 


1 

f Yes. 20% 

No. 80% 

Not sure. 0% 

| 8. Would you be interested in using social networking services as part of a study routine for classes in future? | 

Yes. 47% 

No. 33% 

Not sure. 20% 

9. As a platform for using social networking services as 

part of your study routine for classes, would you prefer an existing system (e.g. 1 

| Facebook, Google+ or Twitter) or a service created specifically for that purpose? 

1 

Existing service. 27% 

New, specialized service. 53% 

Not sure. 20% 

| 10. If using an existing social networking service as part of your study routine for classes, which would you prefer? | 

Existing account on the service. 

New account on the service dedicated to 

No preference. 

50% 

classes. 43% 

7% 

1 1 1. If you answered "Existing account on the service" to the question above, would you be willing to grant an application access to your 1 

| data on the service in order to better allow class interaction to be supported? 

l 

Yes. 44% 

No. 44% 

Not sure. 11% 

| 12. If an experimental system for using social networking services for study was available, would you be willing to use it? | 

Yes. 73% 

No. 13% 

Not sure. 13% 


The students who completed the survey were all enrolled in traditional (not distance learning type) 
Master’s degree -level Computer Science classes. This is reflected in the response to question 1, where the 
respondents overwhelmingly prefer traditional, face-to-face classes. The respondents also overwhelmingly 
study by themselves, however, in responses that are promising for this research, they all use social 
networking services, and they are interested in studying in a group online, if there was a convenient way to 
do so. Most students who study in groups do so in groups of 2 or 3, and a fair number of those who do group 
study using social networking services to interact. The respondents demonstrate a certain amount of 
reluctance to mix personal and study use of social networking as only 27% would prefer an existing service 
over a new specialized service and only 50% would want to use an existing account of a social networking 
service rather than a dedicated one. The remaining questions are promising, in that students do not currently 
integrate social networking services in their study routines, but would be interested in doing so if a system 
was available, at least on an experimental basis. 
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